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The Capability of Virtual Reality to Meet 
Military Requirements 

(RTO MP-54) 


Executive Summary 


PURPOSE 

The purpose of the workshop was to examine military requirements for Virtual Reality technology, 
consider human factors issues in the use of Virtual Reality and review recent research in development 
of Virtual Reality applications to meet military needs. 

SUMMARY 

The workshop was organized into three daylong sessions. The first day focused on military 
applications for Virtual Reality systems and identified particular requirements for Human Factors 
research to meet the requirements. The second day examined Human Factors issues in the use of 
Virtual Reality technology. Presentations discussed sensory interfaces, measures of effectiveness, 
importance of the sensation of presence, and cybersickness. The third day reviewed assessment 
methods and applications research. Speakers reviewed existing or completed Virtual Reality projects 
designed to meet military needs. The papers discussed how the projects overcame human factors 
problems and how their effectiveness was evaluated. Summaries of the paper presentations are 
incorporated in the chairman’s notes for each day of the workshop. 

MAIN CONCLUSIONS 

Virtual Reality technology is of great interest to the military. Requirements for its use encompass a 
wide range of applications including concept development of systems for dismounted combatants, 
mission rehearsal for special operations, training ship handling skills, telerobotics, and practicing 
military medical procedures. Virtual Reality’s success in meeting these needs will be determined by 
the ability of its human-computer interfaces to provide the means necessary to deliver stimuli and 
allow appropriate responses from those using it. These human factors issues were the focus of the 
workshop. Through research on Virtual Reality’s interface technologies and applications, it is clear that 
Virtual Reality has promise for the military, but serious human-computer problems limit its potential. 
In particular, helmet mounted displays need to be improved, cybersickness limits use by a significant 
number of people, haptics and walking interfaces are in their infancy. The workshop pointed to these 
and other areas that require further research and development in order for Virtual Reality to meet its 
potential for the military. 

MAJOR RECOMMENDATIONS 

Research needs to continue on the many human factors issues involved in the use of Virtual Reality to 
meet military requirements. The Virtual Reality technologies are maturing and the feasibility of 
developing cost-effective Virtual Reality based tools is increasing. Research on the usability of this 
technology will enable militaries to be smart buyers. It will ensure that Virtual Reality hardware and 
software is capable of meeting the perceptual, fidelity, transfer of training, and health and safety 
requirements of applications. Military research needs to focus on those issues that are unique to the 
military and not likely to be addressed by other potential users of Virtual Reality. 
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la Capacite de la realite virtuelle a repondre 
aux besoins militaires 

(RTO MP-54) 


Synthese 


OBJET 

Cet atelier a eu pour objet: d’examiner les besoins militaires en matiere de technologies de realite virtuelle, 
de considerer les aspects facteurs humains de la realite virtuelle et de faire une synthese des travaux de 
recherche recents sur le developpement d’applications de realite virtuelle pour satisfaire aux besoins 
militaires. 

RESUME 

L’atelier a ete organise en trois sessions d’une joumee : La premiere journee a porte sur les applications 
militaires des systemes a base de realite virtuelle et a identifie des voies de recherche dans le domaine des 
facteurs humains susceptibles de repondre aux besoins. La deuxieme joumee a ete dediee au probleme des 
facteurs humains dans 1’application des technologies de realite virtuelle. Les communications presentees 
ont traite des interfaces sensorielles, de 1’evaluation de l’efficacite, de 1’importance de la sensation de 
presence et des maladies cybemetiques. La troisieme journee a privilegie les methodes devaluation et la 
recherche en applications. Les conferenciers ont examine les projets existants ou realises dans le domaine 
de la realite virtuelle appliquee aux besoins militaires. Les presentations ont decrit l’approche des 
problemes lies aux facteurs humains et leur resolution, ainsi que les methodes retenues pour 1’evaluation de 
l’efficacite des differents projets. Des resumes des presentations ont ete incorpores aux notes du President 
pour chaque joumee de 1’atelier. 

CONCLUSIONS PRINCIPALES 

Les technologies de realite virtuelle sont d’un grand interet pour les militaires. La demande pour ces 
technologies couvre un large eventail d’applications comprenant le developpement de concepts de systemes 
pour infanterie debarquee, la preparation de missions pour les operations speciales, l’entrainement au 
pilotage des navires, la telerobotique et la formation a l’usage des procedures medicales militaires. La 
capacite de la realite virtuelle a repondre a ces besoins sera determinee par la capacite de ses interfaces 
homme-machine a transmettre les stimuli et a sollicker les reponses appropriees de la part des utilisateurs. 
L’atelier a fait une place privilegiee a ces sujets. Les resultats des recherches entreprises sur les 
technologies d’interface de la realite virtuelle indiquent tres clairement que si la realite virtuelle est 
prometteuse pour les applications militaires, son potentiel se trouve limite par d’importants problemes 
homme-machine. En particulier, les visuels montes sur casque sont a ameliorer, les malaises cybemetiques 
limitent l’acces a ces technologies pour bon nombre de personnes et les interfaces haptiques et ambulatoires 
sont a leur debut. L’atelier a mis en exergue ces domaines entre autres, pour lesquels des travaux de 
recherche et developpement complementaires sont necessaires pour que la realite virtuelle puisse atteindre 
la plenitude de ses possibility militaires. 

RECOMMANDATIONS PRINCIPALES 

II importe de poursuivre les recherches sur l’influence des facteurs humains dans l’application des 
technologies de realite virtuelle aux besoins militaires. Ces technologies viennent a maturite et le 
developpement d’outils bases sur la realite virtuelle a cout abordable semble de plus en plus accessible. Des 
recherches doivent etre entreprises sur la facilite d’utilisation de ces technologies afin de permettre aux 
militaires de s’en approvisionner en connaissance de cause. Ils pourraient ainsi s’assurer que le materiel et 
les logiciels de realite virtuelle sont compatibles avec les exigences en matiere de perception, de fidelite, de 
transfert d’entrainement et d’hygiene et securite des applications. Les chercheurs militaires doivent se 
concentrer sur les questions qui concernent uniquement les applications militaires et qui ne sont pas 
examines par d’autres utilisateurs de la realite virtuelle. 
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Foreword 


Virtual Reality (VR) has the potential to meet needs across a wide range of applications within the military. VR 
military applications include combat training, mission rehearsal, concept development, materiel design, materiel 
testing, medical training, and personnel selection. The ability of VR in meeting requirements in these areas will 
depend upon establishing an effective match between technological capabilities, application requirements and human 
sensory, cognitive and motor capabilities and limits. 

NATO Research Study Group 28 (RSG 28) was established to: 1) identify human factors issues involved in the use of 
VR technology for military purposes; 2) determine the state of knowledge with regard to those issues; and 
3) recommend a research agenda that will address critical questions and enable effective products to be produced to 
meet the military’s needs. RSG 28 has adopted the following definition for its use of the term Virtual Reality: 

Virtual Reality is the experience of being in a synthetic environment and the perceiving and interacting through 
sensors and effectors, actively and passively, with it and the objects in it, as if they were real. Virtual Reality 
technology allows the user to perceive and experience sensory contact and interact dynamically with such contact in 
any or all modalities. 

This report summarizes a workshop conducted by RSG28. The purpose of the workshop was to: 1) discuss issues 
related to military requirements for Virtual Reality; 2) consider human factors issues and research focusing on the 
interfaces between human beings and virtual environments; and 3) review VR military applications research and 
assessment methods. A day of the three-day workshop focused on each of these broad topics. The workshop brought 
together experts in Virtual Reality from ten NATO countries. It was attended by over 60 people and included 30 paper 
presentations. Stephen Goldberg, US served as the overall workshop chair. Neil Hardinge, UK chaired the first day on 
military requirements. Robert Breaux, US chaired the second day on human factors issues, and Peter Werkhoven, The 
Netherlands, chaired the final day, focused on applications research. 

The workshop demonstrated that the military is looking to VR technology to answer a number of its problems from 
dismounted infantry simulations to training ship-handling skills. The technology is improving, and there have been 
VR research success stories. Nevertheless, substantial improvements in VR technology are required before a natural 
man-machine interface is available for many applications, and products are regularly produced that will be 
institutionalized for use by the military. 



Welcome 


Medecin Chef des Services Papin welcomed the participants to the open workshop and outlined the aims of 
the Research Study Group. He provided two definitions of virtual reality (VR) for the workshop to 
consider. The working definition of the RSG is, “Virtual Reality Technology is a multi-dimensional human 
experience which is totally or partially computer generated and can be accepted by those experiencing the 
environment as consistent.” 

General Papin also gave his own definition of VR. In his lectures, he defines VR as having 5 key features 
(the 5 I s), as follows: 

- Information (computer-generated) 

- Interface (using a physical interface) 

- Illusion (of the real-world) 

- Immersive (in a virtual environment) 

- Imagination (imaginative rendering of the real-world by computer-programmers). 

It is expected that VR systems will be used by the military for: 

- Design (to develop man-machine systems suitable for military tasks) 

- Training (to teach new skills & knowledge in complex environments) 

- Rehearsal (to practice skills and procedures in operational settings) 

- Telepresence (i.e. remote interaction with a real object as with robotic sensors) 

- Test & evaluation (e.g. of new equipment or in new environments) 

- Operational analysis (using man in the loop to assess outcomes of military forces, tactics, etc). 

The aim of this group is to study the human factors problems arising from these kinds of applications. Each 
day of the workshop is devoted to examining one of the group’s 3 principal themes — military applications; 
human factors issues; virtual reality technologies. 
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MILITARY REQUIREMENTS FOR VIRTUAL REALITY TECHNOLOGY 

Day Chairman: Neil Hardinge 

DERA Centre for Human Sciences 
MoD 

Famborough, Hants GUI4 6TD 
United Kingdom 

DAY 1: MILITARY APPLICATIONS 

1 Neil Hardinge highlighted the objective of the first day of the workshop - to examine the military applications for 
virtual reality systems and to identify the particular requirements for Human Factors research and related system design 
that these generate. In particular he invited the workshop to consider several key questions: 

What Does The Military Want From VR Technologies? 

• What is special about the context in which that need must be met? 

• Who are the potential users of VR systems? 

• What do you see as a likely solution to the requirement? 

• How will you know if you have met the need? 

He invited the speakers, who represented a broad range of military users - some of them military personnel who defined 
the operational requirement for military systems, some of them civilians associated with the development of VR systems 
- to describe their requirements and to identify the particular problems that they would like this group to address. 

2. The first speakers, Cdr Kvisgaard and Lt Cdr Thomasson from the Royal Norwegian Navy (RNoN), described the 
approach to training design that is being adopted by the Norwegian Naval Training Academy. They described the 
philosophy of training design, the new organisational structure of the Academy and how these would determine the 
RNoN's approach to the design of training systems - especially those that involved simulation and virtual reality. They 
emphasised the importance of specification, cost-effectiveness and evaluation in the Norwegian approach and related this 
to the current acquisition of a navigation and shiphandling simulator using 2D displays. The RNoN’s new approach to 
training needs analysis aims to incorporate the training needs and educational processes more effectively in the technical 
specification. They suggested that the capacity to develop consistent virtual environment databases which capture the 
environment, objects and effects will be the most significant development and could see this providing the basis for a 
new command control and information system. 

3. Dr. Terry Allard from the US Office of Naval Research (ONR) outlined the requirements of the US Navy and Marine 
Corps for immersive VR systems for training and operations. Their compactness and inherent reconfigurability could 
make immersive VR systems an ideal medium for field and shipboard deployable training systems. He described the 
prototype for training shiphandling skills (VESUB) that ONR has funded and which is now being assessed in the USN 
warfare officers school. The Marine Corps identified training for urban warfare as their highest priority and Dr. Allard 
described how the small unit tactical trainer (SUTT) has been developed to help train distributed teams and teams on 
combat deployment in skills such as marksmanship, house to house search, situational awareness, etc. For military 
operations, he indicated how VR systems are used to view the displays and to manipulate remotely operated vehicles 
(ROVs) such as undersea ROVs for mine countermeasures or submarine rescue work. He also described current work to 
explore their value as a display medium for members of ship combat information centres. 

4. As part of the Directorate of Operational Requirements for Training and Light Weapons Systems (Land), Lieutenant 
Colonel Nigel Gallier is directly involved in specifying the requirements for new military systems for the UK Army. 
Mike Kelly - one of the assignment managers whose research programme is driven by those requirements - delivered the 
presentation. The UK's Army Training Strategy has identified a range of training systems in which simulation and VR 
technologies play an essential role. He identified the range of weapons systems for which VR has been selected as the 
appropriate training medium. This creates several challenges: to ensure that the most cost-effective technical solutions 
are developed; to ensure that new training curricula and training approaches are developed which will capitalise on the 
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training potential of these new media; and to develop supporting training features - such as after action review facilities. 
Of particular importance is the development of methods of performance measurement and training effectiveness 
measurement that can ensure that these systems make the essential contribution to operational effectiveness that has been 
assigned to them. 

5. Mike Kelly then described a research programme of his own that addresses the unique problems created by using 
virtual reality to train micro-aperture surgery (MAS, or keyhole surgery). VR is a natural training medium for this kind of 
work. It is normally mediated by television images and touch, and doctors have usually had little or no training 
equipment. Mr. Kelly’s work has provided a task analysis - with surprising results - and has developed some new training 
devices. It has examined the relative contribution of force feedback features and has exploited the availability of organ 
models on the World Wide Web. Perhaps more importantly, the task and the VR system that he is developing lend 
themselves to remote operations in, for example, field hospitals. Originally supported by civilian medical groups, this 
programme is now also being sponsored by the UK Army. 

6. Jan Chervanek described US efforts to integrate the different aspects of (constructive, virtual and live) simulation for 
dismounted soldiers to provide joint training and warfighting solutions. The testbed at Fort Benning provides a synthetic 
environment of urban terrain and links a number of different individual combatants and computer generated forces. He 
described the enhancements that are planned and the technical challenges to be overcome. Although the programme has 
a strong technical focus, it also generates Human Factors issues - such as how to refine the manned virtual interfaces so 
that they realistically represent visual, sound and motion cues and how to model and validate the behaviour of individual 
combatants. 

7. H. A. Warren of DERA, spoke about teleoperation of unmanned ground vehicles (UGVs) for operations such as 
reconnaissance and mine clearance. He outlined the advantages and disdvantages of using UGVs for military tasks. 
UGVs reduce the risk to the human operator, thereby allowing for bolder concepts of operation. In addition, the UGV 
itself is not prone to fatigue, boredom, or stress, all of which can adversely affect mission success. They also aid in the 
goal of manpower reduction. However, they are also expensive, lack user confidence and acceptance, are an increased 
logistical burdem, and are less adaptable and flexible than human operators. Mr. Warren then presented an overview of 
the engineering changes made to the AVRE for remote operation. 

8. Mr. P. Gibson of DERA outlined telepresence and issues in using remotely operated vehicles. He noted that fully 
immersive VR might not be needed as simple cues can produce a mental framework that allows the user to predict future 
events. After presenting a background on what type of sensory data is needed for complete telepresence, he again argued 
that such complexity is not needed, as the user’s imagination will fill in any missing information. This more minimalist 
perspective is particularly relevant given the expense and difficulty in implementing some types of teleoperational 
systems. Therefore, research should also focus on what cues are necessary to aid the imagination. He then applied these 
ideas to a remote control tank system. 

9. LTC Stone described the usefulness of the Virtual Reality Markup Language (VRML) for the Functional Description 
of the Battlespace (FDB), which is the Army domain modeled for simulation development, to teach concepts needed in 
complex domain modeling. The FDB data is comprised of four main categories: 1) Human Characteristics, for example, 
behaviors and capabilities; 2) Systems and Materiel, including the impact of supply and maintenance operations; 3) 
Physical Environment, including natural and manmade conditions and their dynamic changes; and 4) Organizations, 
covering cognitive processes of command and control 

10. Tom Mastaglio provided a completely different perspective to the day’s proceedings by discussing the factors that 
make VR technology and HF research important for commercial applications. For example, he now works for a private 
agency that seeks to transfer military technology to commercial applications. 

Using VR for system prototyping was not discussed that day but it was the subject of a presentation on Day 3. There 
were no presentations on Air systems requirements on the first day. There is a vigorous work programme in this area - 
especially in the development of helmet mounted displays, augmented reality presentations and control interfaces to 
reduce pilot workload. The proceedings of the 1996 meeting of the previous AGARD Panel (1) provide a good insight 
into that work, and (2) it is hoped that increased interchange in the future will advance the work of the interest groups. 
Members of the aerospace community were represented in the audience and were well represented in the first RSG28 
workshop. 
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DISCUSSION 

The chairman put a number of questions before the speakers and audience to determine their views: 

•What special advantages might VR provide? 

•What are the costs? 

•What are the benefits? 

•What barriers - human factors, operational issues, technological limitations, etc - will users need to 
overcome? 

•How can the R&D community help facilitate the development of effective VR systems? 




1-1 


INVESTMENT EVALUATION OF TECHNOLOGY-BASED TRAINING 
SYSTEMS: TRAINING NEEDS ANALYSIS IN 
THE ROYAL NORWEGIAN NAVY 

E. W. Berghout 

Department of Information Sciences 
Delft University of Technology 
Julianalaan, 134 
Postbus 5 

2600 AG Delft, The Netherlands 

C. Svendsen 
Siemens 

I. W. Thomassen 
HNoMS Tordenskjold 

ABSTRACT 

This paper describes the results of a research done in the Royal Norwegian Navy into the decision structure of the 
evaluation process of investing in technology-based training systems. A comprehensive methodology was developed, 
that evaluates technology-based training systems as information systems. The evaluation method can reveal cost driving 
training objectives and make sure focus on the strategic value of the information system components is maintained. 

INTRODUCTION 

The Royal Norwegian Navy (RNoN) is currently evaluating investments in new operational equipment. Investments in 
technology-based training systems are a part of these investments. The RNoN has developed a cost/benefit analysis 
method that is particularly suited to evaluate technology-based training systems. The investment evaluations conducted 
so far have merely focused on the training needs. 

A more comprehensive methodology has now been developed, that evaluates technology-based training systems as 
information systems, and also takes account of their technological and life-cycle consequences. The evaluation method 
was developed by extensive literature investigation and case studies. The outcome of the case studies was used to 
improve the decision structure of the evaluation method. This method was developed in a research co-operation between 
Delft University of Technology in The Netherlands and the Naval Training Establishment HNoMS Tordenskjold in 
Norway. 

TECHNOLOGY-BASED TRAINING SYSTEMS 

This term can best be defined by other definitions. An information system can be defined as a collection of software, 
hardware, people, procedures, and data (Brussaard, 1994). Further, an instructional system can be defined as an 
arrangement of resources and procedures used to promote training (Gagne, Briggs, & Wagner, 1992). In this research the 
context of information systems was within the instructional system where information systems are used to promote 
training. Training is the learning or acquisition of skills, knowledge and attitudes in order to enhance performance at a 
given task (Bumiston & Tayler, 1995). 

In this paper the term technology-based training implies the use of information systems in any part of the instructional 
system. To emphasise that an information system is used to promote training, the term chosen is technology-based 
training system. This implies that technology-based training systems must also consist of the five information system 
components. 


Paper presented at the RTO HFM Workshop on u The Capability of Virtual Reality to Meet Military 
Requirements ”, held in Orlando, USA5-9 December 1997, and published in RTO MP-54. 
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Case 1: Communication Trainer 

The communication trainer was used in the training of internal and external operational communication procedures. The 
communication trainer is an integrated part of the existing technology-based training systems at the Naval Training 
Establishment HNoMS Tordenskjold. The main objectives of the communication trainer are: to take care of all types of 
training, covering procedures and plans for using existing operational communication equipment. Secondly, internal 
communication between trainees and trainees/instructor. The third objective was to provide a standard platform for voice 
communication for existing and future technology-based training systems within the RNoN. 

Case 2: Scenario Management System 

The scenario management system aims to improve the instructor’s working environment by providing the instructor with 
the required tools for supporting the training process. The system’s main objectives were to enable re-design of the 
instructor function to reduce the number of instructors involved in the training process. Further, it was intended to 
develop a standard instructor workstation and a prototype of an interconnection device for the Navy’s future technology- 
based training systems. 

THE EVALUATION METHOD 

The idea behind the evaluation method 1 described in this paper is to provide a structured approach to the decision process 
of investing in technology-based training systems. An analysis of training needs , information system components and 
costs are required before a decision can be made on what investment strategy to pursue. These analyses are important to 
prevent overlooking unfavourable consequences in the initial enthusiastic planning phase. 

In the context of evaluation of technology-based training systems the term value is defined as the sum of associated costs, 
benefits and risks of the desired competence level and its support technology. In other words, value can be structured into 
the three components of (1) training domain evaluation, (2) technology domain evaluation and (3) cost evaluation. These 
evaluations can be done by means of the method outlined in Figure 1. Consequently, to determine the value of different 
investment strategies requires a structured approach by the RNoN organisation. The different elements of the method 
(Figure 1) are described in the following sections. 


Identify System & Search Space 


When a particular problem is perceived, it is of course important to be sure that an associated training need exists. It may 
be that an apparent need can be satisfied more economically in some other way, for example by a change in the 
organisation or the introduction of a different personnel policy (Akerjordet, 1993; Patrick, 1992). 
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Figure 1. Method for investment evaluation of technology-based training systems (Svendsen, 1996) 


1 The term often used for the evaluation process of technology-based training systems is: training needs analysis (TNA) (HMSO 247, 1992; RNoN 
NSST, 1995; Gramson, 1995). 
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Identification of needs, problems, and opportunities are done in order to trace valuable technology-based training system 
solutions for the organisation (van Reeken, 1994). The identification-process can be carried out on the basis of the 
information system strategy formulation principle described by Earl (1989). It can be done in three complementary ways, 
top-down, bottom-up, and inside-out. This process can also be used to evaluate technology-based training system 
investment strategies. The way this identification process is performed is by means of an approach strategy. The three 
main approach strategies are; (1) clarification of organisational goals, (2) evaluation of existing investments and (3) 
innovation by application of new technology. A fourth approach strategy is to combine the three strategies in various 
ways. The strategies of differentiation and focus (Porter, 1985) can be used to further improve the scope of the evaluation 
approach. 

When a discrepancy in the form of a need, problem, or opportunity is identified, further analysis can be undertaken. For 
this analysis, a mission statement or a project goal must be defined. A mission statement provides a long-term continuity, 
because it gives the organisation and its employees direction and purpose (Ishoy & Swan, 1992). 

Task Analysis 

Task analysis is an analysis of the final task for which training is to occur. In other words, it derives the main elements of 
the training content (Patrick, 1992). Without the task analysis, there can be no guarantee that the training given will not 
be deficient in some way (ENTWG/TT-PUB 2, 1983). In addition to the analysis of the tasks to be trained, the task 
analysis must also include the instructor tasks. From the Cases the task space also includes the instructor tasks. By 
including the instructor tasks in the task space of the planned technology-based training system, a more complete picture 
of the system can be created. 

A job is the duties, tasks and sub-tasks performed by one individual (ENTWG/TT-PUB 1, 1990). Where the term task is 
any set of activities occurring at about the same time sharing the same common purpose that is recognisable by the task 
performer (Patrick, 1992). 
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Figure 2. The task hierarchy of the scenario management system 


The task can be broken down into sub-tasks, which are parts of a task that may not stand alone. When dealing with task 
analysis it is important to know what is meant by it, and how it is achieved and carried out. 

The task analysis technique used in the evaluation method was based on an object-oriented approach (Bots, 1989). This 
technique is a combination of the critical incident technique (Flangan, 1954), hierarchical task analysis and Petri nets 
(Kirwan & Einsworth, 1992). The advantage of this object-oriented technique is its descriptive as well as its prescriptive 
way of illustrating tasks and sub-tasks (Bots, 1989). 

Figures 2 and 3 give a high level description of the tasks of the scenario management system of Case 2. Figure 2 
illustrates the task space. Figure 3 illustrates the relationship between the different sub-tasks. This object-oriented task 
analysis technique may help to reduce differences in terminology and create a common ground for understanding 
evaluating technology-based training systems. 

Audience Analysis 

Analysis of the trainees will vary from an analysis of main features and down to a detailed analysis of the individual 
trainee (UD 3-2, 1984). The nature of the trainees must be determined and their skills assessed (ENTWG/TT-PUB 2, 
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1983). The first decision to be made, is how much information is necessary for an adequate audience identification or 
analysis. An analogy to conduct an audience analysis is found within product marketing (Moller & McDermott, 1992). 
Following this idea, the trainees or target audience can be divided into different clusters (Kotler, 1994). Clustering of the 
target audience can be done after different criteria. 

In Case 1 the target audience was clustered after its respective organisations. Some of these organisations were Navy and 
Coast Guard vessels, and coastal fortresses. These organisations are situated at different locations which implies a target 
audience scattered 




Figure 3. The scenario management task. 

over a geographical area. Considerations about the access to different clusters of the target audience must be done when 
evaluating technology-based training systems. 

Further, the target audience can also be clustered after their respective training courses and career programmes. When the 
clustering of the target audience is done the assessment of the trainees characteristics can be performed. 

The target audience analysis will account for the users of the system for which training is to take place. It is necessary to 
know at what stage the trainees enter the training process (i.e. what skills, knowledge and attitudes they possess prior to 
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training Bumiston & Tayler, 1995). This is important because a technology-based training system can be used for both 
initial training and continuation training (NIAG(94)SG/45-R4, 1994). In the Royal Norwegian Navy technology-based 
training systems were applied on three main levels of training. These were the training of (1) individual, (2) team and (3) 
command operations (RNoN NSST, 1995). 

Competence Contribution 

Competence is the skills, knowledge and attitudes required to perform an operational task. Different technology-based 
training systems may differ in meeting operational competence requirements. To evaluate this competence contribution, 
training objectives have to be defined. A training objective can be seen as an expression that emphasizes what the 
technology-based training system must be able to support for training to occur. Subsequently, these training objectives 
are statements of new competencies the trainee should possess at the completion of training to meet operational 
competence requirements (Russel, Molenda, & Heinich, 1993). 

To determine the training objectives structured guidelines are available. One way of detailing training objectives is by 
means of the convention called the five-component objective (Gagne et al 1992). The five-component training objective 
specifies elements such as the situation, required competence level, specific actions, tools and constraints (Gagne, 1992). 
Well-defined training objectives are essential when determining the training method to be used. 

The training objectives can also be used to gain information on the training system’s temporal aspects. In Case 1 the 
following questions were asked. In what time perspective are the training of these tasks valid? Further, will changes in 
the operational organisations in the near future have any impact on the training objectives? Temporal analysis is 
important for the future effectiveness of the technology-based training system. 

In addition to the training objectives in Case 1, instructor objectives were defined. The instructor objectives were also a 
natural element in Case 2, to pinpoint the instructor’s needs in the training process. The training and instructor objectives 
are used to evaluate different technology-based training system investment strategies. The different strategies may range 
from simple to complex systems solutions. 

Training Commitment 


Organisational readiness analysis is performed in the Training Commitment stage of the evaluation method. It examines 
the commitment by the subject matter responsible organisation to implement the proposed technology-based training 
system into its training programmes. 

In Case 1 an important issue was improving the utilisation of existing technology-based training systems within the 
RNoN. An aspect of this improved utilisation is how the planned technology-based training system fits into the existing 
and future training courses and programmes. Decisions on including the system in the different training programmes 
were up to the subject matter responsible organisation and not the service provider department. For the subject matter 
responsible organisation it can be a question of convenience and practicality before the issue of training effectiveness. 
Training effectiveness is the best utilisation of resources available in bringing the trainees or target audience up to the 
required standard (ENTWG/TT-PUB 1, 1990). 

Applications 

This stage covers the transformation of revealed training and instructor objectives into an environment for application 
delivery solutions in the form of technology-based training systems. 

The man-machine interface (MMI) is the way the trainees and instructors communicate with the technology-based 
training system. Therefore it is important that the MMI provides the necessary cues for training to occur (Bumiston & 
Taylor, 1995). In the two Cases the MMI were windows-based. In Case 1 the MMI emulated the functionality and 
presented the image of the operational communication equipment. The purpose-built hardware implemented were the 
different types of telephones resembling the operational parent equipment. The evaluation of man-machine interface 
becomes more difficult when considering more complex technology-based training system solutions. Here in addition the 
MMI can be purpose-built hardware to replicate the operational equipment, as in Case 1. The reproduction of 
kinaesthetic cues, visual images and audio must also be included. 
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A technology-based training system may consist of different software modules. In both Cases different system modules 
were defined such as database management systems (DBMS), scenario generation and simulated entities. Other system 
modules used in a technology-based training system can be geographical information systems (GIS), and modelling and 
animation tools. In addition to these software modules the operating system software must also be evaluated. In the two 
Cases both client and server operating system software were evaluated. 

Data Needs & Requirements 

Data forms the basis of every information system. The most important issue here is to determine the data structure and 
the most important data-elements (Eilers, Dijkstra & Hoven, 1991). 

In Case 2 a database design was developed to reveal the data foundations required by the system. This database 
incorporated elements such as scenario descriptions, configuration and model parameters, charts organisation and trainee 
information. Other data elements considered were charts and terrain information available on standard data formats. 
Respectively formats such as DX-90, digital terrain elevation data (DTED) and digital feature analysis data (DFAD) 
(RNoNNSST, 1995). 

In Case 1 and Case 2 evaluation of data elements that must be exchanged with other existing or planned technology- 
based training systems were also undertaken. 

Computing & Communication 

It is necessary to perform analysis of the required computing architecture of the technology-based training system. This 
provides an insight into the ability of a particular technology-based training system’s architecture to meet performance 
requirements. The computing architecture of technology-based training systems may range from a collection of elements 
such as image generators, audio systems, kinaesthetic control systems, specific-purpose built components and scenario 
management work stations, to personal computers in a network or stand-alone solutions. 

In the Cases, both the internal communication network and the external integration with other technology-based training 
systems were evaluated. In Case 2 the system operated in a distributed environment with a local area network as central 
communication media. In addition it was supposed to exchange information with other systems by means of a distributed 
interactive simulation (DlS)-gateway. 

People & Procedures 

To implement the technology-based training system a set of procedures are required. These procedures or management 
activities differ with the life-cycle state of the system (Looijen, 1991). In both Cases management procedures were 
outlined. 

In Case 1 the management of the technology-based training system was to be carried out with the existing staffing level. 
Human recourse evaluations are required for the fulfilment of these management procedures which are necessary to keep 
the training system operational. The method supports the evaluation of required management procedures and staffing 
considerations for technology-based training systems. 

Cost Analysis 

To gain insight into the financial aspects of technology-based training system cost analysis must be undertaken. From the 
Cases the initial investment cost was identified. Further in Case 1 the life-cycle cost was estimated for the different 
investment strategies. The initial costs can be divided into two categories (Lien & Bjorn, 1995). First all the costs related 
to the planning and accomplishment of the project and then the second category covering all the costs concerning the 
procurement of software, data, hardware and facilities. The life-cycle costs regards all costs associated with the training 
system’s maintenance and management activities. 
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The effectiveness of a technology-based training system is whether or not it meets well defined objectives that are based 
on the needs and constraints of its ownership organisation. The method suggested in this paper can be a helpful means to 
evaluate this effectiveness. 

By means of the evaluation method different investment strategies for technology-based training systems can be outlined. 
These different investment strategies can be illustrated by means of an evaluation matrix (Berghout, 1994). The vertical 
and horizontal axes portray, respectively, the training and technology value of the investment strategy. The area of the 
circle reflects the initial and life-cycle cost. 



Figure 4. The investment strategy evaluation matrix (Berghout, 1994). 

The training value of a technology-based training system is its effectiveness, efficiency and relevance to the operational 
task, as perceived by its users. The technology value is the strategic alignment of the information system components 
with organisational information plans. 

Reasons for a more formal approach to investment evaluations of technology-based training systems are: 

• The large sums of capital that are often involved in procurement of technology-based training systems. 

• Technology-based training systems often have significant consequences for other technology investments, hence 
compete with other investment opportunities (Hares & Royle, 1994; Berghout, 1994). 

• Formal approaches create a common ground for communication between the different parties involved in the 
decision process. 

• Training needs and their technical solutions need to be co-ordinated. 

• Increased awareness exists on training objectives and its relevance to the operational situation. 

• The initial enthusiasm regarding new products tempt decision makers to overlook possible negative consequences. 

The investment analysis technique that has been presented in this paper seems to provide a comprehensive evaluation of 
technology-based training systems. 
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US NAVY AND MARINE CORPS REQUIREMENTS AND CHALLENGES: 
VIRTUAL ENVIRONMENT AND COMPONENT TECHNOLOGIES 

Dr. Terry Allard 
Office of Naval Research 
Arlington, VA, USA 

The US Navy and Marine Corps have clear requirements for virtual environment (VE) systems and related component 
technologies for both training and operations. VE systems have been defined broadly to include any computer-based 
representation of an artificial or real world regardless of the medium of display. A more restricted definition includes 
only those systems that rely on some form of immersive display where the user experiences and interacts directly with 
the virtual world as if they were part of the simulation. This paper addresses requirements and technical issues related 
to the latter category of immersive VE systems. Discussion will be limited to land, sea and subsurface applications 
without in-depth treatment of aviation and targeting requirements. 

Immersive VE systems are characterized by multiple sensory modalities and direct interaction. VE displays can 
include head-mounted visual displays, which are yoked to head position producing the illusion of fully immersive 360- 
degree immersion. Head-mounted visual displays can be monocular or “see-through permitting the observer to 
integrate real surroundings with virtual objects and displays. Another version of visual immersion uses projection 
displays on a small enclosed space allowing free head movements and natural exploration of the surrounding virtual 
space. The sense of physical presence created by immersive visual displays may be augmented by spatialized (3-D) 
audio, which can simulate the acoustic filtering of the head, pinnae and facial features as well as virtual room 
acoustics. Spatialized audio creates the perception of sound sources well-localized outside the head rather than 
between the ears as is usually the case with headphone presentation of sound. This expansion of the perceived 
workspace of the operator takes advantage of the human ability to attend to multiple spatially discrete information 
sources. As we shall see later, both virtual visual and audio have the potential for greatly increasing the personal or 
shared workspace of individual operators. 

In addition to the better known visual and auditory virtual displays, there have been significant advances in the 
rendering and display of virtual haptic representations (Tan, Durlach, Beauregard, & Srinivasan, 1995; Srinivasan and 
Durlach; http://touchlab.mit.edu/). Haptic perception combines tactile (touch) and force-feedback (proprioceptive) and 
is usually associated with active motor exploration of the objects being explored. Haptic displays can be used as 
direct manipulation interfaces and together with automatic speech recognition help to make interactions with 
immersive VEs truly direct and natural. Immersive VEs can be defined by any or all of these features but most 
commonly are associated with temporally synchronized and spatially registered virtual visual and auditory displays. 

VE systems should be considered as potentially augmenting or replacing dedicated simulator technology, especially 
with respect to training applications. Conventional simulators are dedicated hardware and software systems, often 
costing millions of dollars to acquire and maintain and requiring large spaces to house the systems physically. These 
size and cost constraints often limit the number of available systems and can only provide limited accessibility for a 
broad range of users. While these systems are often magnificent in their fidelity and potential effectiveness, more 
affordable and accessible simulation-based training systems are required to reach a larger training market. 
Reconfigurable computer-based training systems including VE are available to a larger market because of their low 
cost and inherent deployability. Such lower fidelity systems can be used to optimize limited time spent in more 
expensive dedicated simulators and may, in some cases, actually be as capable or more capable in terms of training 
effectiveness. Rapid advances in VE display technologies and computing power coupled with decreasing system costs 
will continue to improve the fidelity and usability of interactive VE systems. 

VE systems have many advantages over conventional simulator systems for training and mission rehearsal. Unlike 
conventional dedicated simulators, most immersive virtual environment systems are compact and have the potential to 
be reconfigurable in software. This means that a single virtual environment interface could be used for multiple 
training domains as diverse as shiphandling, maintenance and flight operations. 

This reconfigurability makes VE an affordable alternative. Both the compactness and inherent affordability of VE 
systems makes larger production runs possible and increases the potential for large-scale deployment in the classroom 
and in the field. The resultant increase in availability of VE systems makes training systems more accessible to more 


* Currently at: Human Factors Research & Technology, NASA Ames Research Center, MS262-11, Moffet Field, 
CA 94035-1000, USA. 
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potential students and could optimize the limited training time spent in conventional and dedicated simulators. 
Technical challenges to field and shipboard deployment of immersive VE training systems must be addressed quickly 
as there is a groundswell of requests to deploy these systems as quickly as possible. 

Operational uses for immersive VE systems outside of training and mission rehearsal are a longer-term developmental 
challenge. There seems to be some potential for VE as a control system for remotely-operated vehicles (ROVs) 
among other teleoperation applications which require a sense of remote spatial situation awareness and maneuvering 
skills. Performance aids have been discovered as training approaches for pilots of undersea ROVs where close-in 
maneuvering is a critical control feature and where current control interfaces are not well designed from a human 
usability perspective. Non-immersive volumetric displays such as the Virtual Reality Responsive Workbench 
(Rosenblum, Naval Research Laboratory) show great promise as a medium for operational planning and distributed 
coordination and could be augmented or replaced by personal head-mounted immersive VE technologies. 

More near term operational opportunities are VE component technologies that together provide immersive VE 
experiences. Spatialized audio and heads-up or monocular displays have immediate value to the design of optimal 
human-computer interfaces by increasing the potential display real estate available to the designer and the user. We 
shall see later that these component technologies are already making inroads into operational Navy systems such as the 
advanced multi-modal watchstations and distributed integrated command environment (ICE) concepts in the SC21 
ship design program (21st Century Surface Combatant; http://sc21.crane.navy.mil). 

SIMULATION-BASED TRAINING: A SEAMLESS CONTINUUM 

Current Navy and Marine Corps training doctrine is to “train as we fight.” This philosophy leads to a scenario-based 
training approach that lets our warfighters experience some measure of the chaos and unpredictability of combat 
before they are thrown into the real thing. Scenario-based training allows a full range of experience to the warfighter 
from mundane to high-pressured mission rehearsal and allows the user to experience dangerous situations and attempt 
potentially high-risk solutions and strategies that would not be feasible in real-life training exercises. 

Another implication of the “train as we fight” approach is that it is better to train in the actual operational environment 
with the same people and the same equipment than to train in some remote location on mockups or other degraded 
facsimile of the operational environment. According to this reasoning, the farther away the training experience is from 
the actual operational experience, the less effective will be the transfer of training to the real operational world. 

These considerations have lead the Navy to emphasize embedded training systems for almost all manned shipboard 
systems. Onboard scenario-based embedded training systems are well placed to provide training in an operational 
context that facilitates training transfer. Because of their inherently deployable nature, embedded training systems can 
also provide just-in-time refresher training or mission rehearsal as needed driven by emerging mission requirements. 
Embedded training systems must also be designed to diagnose training needs of individuals and teams through 
computer-based monitoring. This information has the dual advantage of providing feedback to shipboard instructors 
as well as forming the quantitative foundation for a performance-based measure of personnel readiness required in a 
reduced manning force. Timely information about operational skills is absolutely essential for intelligent management 
and optimization of limited personnel resources. Shipboard embedded training systems do have the potential to 
greatly reduce the shore-based training infrastructure and may provide some long-term affordability benefits. 

Embedded training approaches do have a potential downside. Operational requirements and training requirements can 
often conflict restricting the availability of training opportunities. This restricted availability is compounded when the 
training exercises require multiple team members, making it even less likely that everybody will be available without 
taking the operational systems off-line. Shore-based training resources are being reduced to save money and to 
streamline the training pipeline. Given this trend, additional stand-alone training capabilities are required to augment 
and support the increasing dependence on embedded training systems. More pierside and shipboard training 
opportunities are needed. 

Stand-alone simulation-based training systems can complete a seamless training continuum from shore-based 
schoolhouse and pierside training to onboard systems including stand-alone and embedded systems. Simulation-based 
systems can emulate the operational environment including both equipment and virtual team members and adversaries. 
To the extent they are successful at emulating operational systems in terms of functional and physical fidelity, they 
will have similar benefits to embedded systems. Hypothetically, it should not matter to a student fully immersed in a 
stand-alone training system whether the system is deployed in the schoolhouse, pierside or onboard. Stand-alone 
onboard systems will have the additional advantage of allowing accessible just-in-time training tailored to the needs of 
the individual without conflicting with the operational requirements of the ship. 
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Stand-alone simulation-based training can be delivered at multiple levels of physical and functional fidelity, from 
interactive courseware on CD-ROM to fully immersive virtual environments. These computer-based systems have the 
advantage of being reconfigurable in software so that a single delivery system can serve a broad range of training 
domains. Critical task analyses and studies of training effectiveness must focus on the conditions under which a 
specific training delivery approach is warranted and cost effective. As display and interaction technology improves 
and costs decrease, fully immersive VE displays will become commonplace and affordable. It is important to begin to 
define the acceptable boundary conditions for use of immersive VE systems deployed ashore and afloat. Critical 
factors to consider include effects of platform motion, field of view, update delays and duration of immersion on 
physical symptoms of sensorimotor adaptation. 

NAVAL SHIPHANDLING REQUIREMENTS 

The US Naval Air Warfare Center, Training Systems Division (NAWC-TSD), with the sponsorship of the Office of 
Naval Research (ONR) has been addressing the needs of the surface and subsurface Navy in shiphandling and harbor 
piloting with immersive VE training systems. In the submarine community, harbor piloting is an infrequent but 
critical mission function with few training opportunities. While there is an existing below-decks team trainer 
(Submarine Piloting and Navigation or SPAN), conventional simulator technology was considered too expensive to 
provide an immersive visualization capability for the critical leader of the submarine navigation team, the officer of 
the deck (OOD). The ONR-supported 6.2 exploratory development program in virtual environment training 
technology (VETT) produced a prototype submarine shiphandling trainer that has transitioned to the 6.3 VESUB 
program (Hays) and is currently in its third year of funding. 

This successful development of a VE training approach to submarine shiphandling has led the Office of Naval 
Research, NAWC-TSD and the Surface Warfare Officers School (SWOS) to evaluate the related training requirements 
of the surface warfare community. SWOS has identified the need to augment or replace existing dedicated simulator 
training capability that costs the Navy in excess of $2M/year to maintain and suffers from the drawbacks in 
availability outlined above. The possibility of compact and deployable VE systems is consistent with SWOS’ vision 
of a seamless training continuum that could include other shore-base facilities where a stand-alone system could help 
sailors maintain their shiphandling skills while on extended shore duty or during sea-going deployments. Department 
Head training at SWOS has been identified as the primary target of the new VE shiphandling systems. Critical task 
analyses and experiments are underway at SWOS, NAWC-TSD, MIT and elsewhere to define specific training 
objectives, curriculum and instructional approaches that enhance the training effectiveness and usability of proposed 
simulation-based systems. Operational challenges being addressed in the near term include underway replenishment 
(UNREP), plane guard and harbor transit. Technical challenges include understanding “seaman’s eye” at a perceptual 
and cognitive level, ship hydrodynamics and interactions as well as the human-computer interaction (HCI) design 
guidelines for the development of usable VE systems. Of particular interest is the value added of immersive VE 
interaction versus more conventional desktop approaches. 

URBAN WARFARE REQUIREMENTS 

The US Marine Corps has identified training for urban warfare as their highest priority. Urban warfare is the 
bloodiest, most casualty-intensive warfighting challenge facing our military services today and training the high-level 
skills required to be effective in these difficult combat scenarios is of critical importance. The Marines are strong 
proponents of the simulation-based approach to combat training and have instituted a series of large scale distributed 
training exercises (Hunter Warrior, Urban Warrior and Capable Warrior) to evaluate new methods of distributed team 
training and operations, with a special focus on small unit operations. 


One example of distributed urban warfare training is a Marine Corps project being developed at NAWC-TSD called 
Small Unit Tactical Training (SUTT). Training objectives and HCI technical challenges of this effort include 
marksmanship, house-to-house search, person-to-person nonverbal communications and global spatial situation 
awareness of individuals and team members for immediate search planning and execution. Ultimately these training 
modules will allow distributed team training across sites that are widely distributed geographically. The Marines 
would like to see this capability developed for shipboard mission rehearsal while underway on combat deployment. 
Similar training and mission rehearsal capability would be useful for Special Operations forces. 

The current SUTT simulation approach uses a single screen back projection technique with no head-mounted display 
and a relatively artificial locomotion interface and speech input system. One advantage of this system is that it allows 
the integration of marksmanship to the urban warfare training objectives. Potential drawbacks to this approach are that 
spatial situation awareness must be reconstructed in a very unnatural way that could impede the development of 
individual and team tactical skills. With the single screen projection approach, the user can only see into the virtual 
world through a single screen like a picture window in front of the viewer. In order to look to the side, users must 
artificially swing the world around to the front by turning their head and body more than 45 degrees to the desired 
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viewing direction. This unnatural and perhaps maladaptive motor behavior conflicts with basic proprioceptive and 
vestibular sensory input and creates an inconsistent mapping of the local virtual environment orientation with the 
global coordinates of the real world the user can still see and feel around him. At minimum, the control and display 
interface is a distraction and additional information overhead for the user. At worst, such a display interface will 
disrupt the very spatial orientation and situation awareness that the system is meant to train. 

There are a number of possible technical approaches to the SUTT spatial situation awareness training objectives. 
Immersive visual displays including head-mounted displays (HMDs) and immersive room environments (e.g., CAVE) 
would provide a complete 360 degree perspective that would not produce conflicting spatial awareness cues to the 
user. The HMD solution has the added benefit of compactness and deployability but presents a further technical 
challenge when the marksmanship requirement is added to the training requirements. The immersive room 
environment potentially could address the marksmanship requirement as well and new technologies could be 
developed to reduce the footprint of the immersive room. In addition, researchers at the US Naval Research 
Laboratory in Washington, DC, are developing and evaluating a more natural and easy to use locomotion interface that 
would reduce the information overhead required to use all of the simulation-based training concepts described above 
(Templeman). 

REMOTELY OPERATED VEHICLE REQUIREMENTS 

Remotely Operated Vehicles (ROVs) are unmanned air, land or sea vehicles that are teleoperated by human users at a 
remote location. The primary benefit of teleoperation is the projection of human skill or intelligence into a potentially 
hazardous remote environment. Some technical challenges for ROV operation are developing a spatial awareness of 
the remote environment of the vehicle and developing a vehicle control capability that includes maneuvering and, in 
some cases, telemanipulation. Virtual environment simulation for teleoperation can provide a seamless 360 degree 
reconstruction of the remote environment that could actually be an improvement over having a human observer 
directly at the site. For example, undersea ROVs are often used at extreme pressures in cold, dark, silty ocean 
environments that would be extremely hazardous or impossible for human divers. Multi-sensor fusion (visual, IR, 
sonar, force feedback) can support reconstruction of the remote scene that would be better that the limited visual 
inputs of a diver on the site. 

Undersea ROVs address military requirements in mine countermeasures and submarine rescue as well as commercial 
and military needs in search, salvage and inspection (hulls, undersea oil rigs and pipelines, etc.). There is a scarcity of 
experienced Navy ROV pilots and a large turnover of competent pilots makes ROV pilot training a major training 
requirement for this community. 

A consortium of industry and academia researchers (Imetrix Inc., Boeing, GTE, and MIT) supported by ONR is 
addressing this requirement in the TRANSoM (Training in Remote Sensing and Manipulation) project. The 
TRANSoM team has developed a stand-alone maneuvering skills trainer for pilots of undersea ROVs using on-line 
intelligent coaching that tailors instruction to fit the needs of individual students. Current efforts are looking at 
extending the training to spatial situation awareness (including tether management) and the relative value added of 
immersive versus conventional desktop displays. 

VE COMPONENT TECHNOLOGIES 

Many of the technologies developed for immersive VE systems will find near term application as non-immersive 
augmentations to existing or planned operational environments. As discussed earlier, head-mounted visual and audio 
displays can greatly extend the perceived workspace of an operator, taking advantage of the human capability to attend 
to multiple information sources that are spatially distinct. This expanded display space can be used in the design of 
consoles that optimize information management and workload. Volumetric information displays can improve global 
spatial awareness and might be valuable to provide a common picture for distributed interactions among 
geographically separated team members, reducing the communications overhead of team interactions. Personal 
portable displays can provide decision support where and when it is most needed. For example, “see-through” or 
monocular display approaches might be able to superimpose technical schematics on a specific mechanical system 
during routine maintenance, diagnosis and repair. 

One example of VE component technology aiding operational design is in the advanced multi-modal watchstation 
being developed by Dr. Glenn Osga and colleagues for the ONR-funded SC21 Manning Affordability initiative. One 
objective of this program is to reduce AEGIS Combat Information Center manning by at least 50% through the use of 
human-centered design principles. The SC21 program office has conservatively estimated that success in this program 
objective will reduce at least 23 billets per ship providing a cost avoidance of more than $1.3B over the expected life 
cycle of the ship class. Osga and colleagues plan on exploiting head-worn displays and spatialized audio to expand the 
perceived workspace of the combat systems decision-makers. The multi-modal watchstation design can be used for 
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other knowledge domains across the ship including damage control and maintenance decision-making and in the 
design of human-machine interfaces for other ship classes using similar human-centered design principles and VE 
component technologies. 

A larger vision for VE component technologies is found in the DD21 Integrated Command Environments of the SC21 
program office (http://sc21.crane.navy.mil). In this vision of cooperative distributed decision-making, multiple 
individual decision-makers will communicate with the use of personal display systems similar to the advanced multi¬ 
modal watchstation. It remains to be seen whether fully immersive virtual reality simulations will ultimately play a 
role in recreating a shared decision-making environment across geographically distributed teams. 

SUMMARY 

Current military requirements for simulation-based training and optimized personal displays provide a compelling 
opportunity for implementation of virtual environment technologies. Near term applications for VE include land, sea 
and air training systems that will ultimately be used in a geographically distributed fashion for team training, mission 
planning and mission rehearsal. Stand-alone training systems will complete a seamless simulation-based training 
continuum from the schoolhouse to pierside and shipboard, augmenting the current Navy and Marine Corps emphasis 
on embedded training in operational systems. Virtual environment training systems have the great advantages of 
compactness, deployability, software reconfigurability and affordability when compared to conventional dedicated 
simulator systems. These advantages will lead to more widespread and available training capability. Careful task 
analyses, human-centered design principles and methods and better performance metrics will be needed in order to 
meet these emerging requirements. A continuing challenge for VE training systems is to demonstrate the value added 
of immersive versus more conventional desktop delivery systems. Further work needs to be done to evaluate the 
potential for shipboard VE systems and safety guidelines for their use. However, with the rapid advances in display 
technologies and computing power in industry today, we can expect VE technologies to continue to grow in value and 
availability as we enter the next century. 
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TRAINING MINIMAL ACCESS SURGERY SKILLS WITHIN A VIRTUAL 

ENVIRONMENT 

M. Kelly * 

Centre for Human Sciences, DERA, Famborough, GUM OLX, U.K 

ABSTRACT 

A training system for Minimal Access Surgery (MAS) has been developed by the Centre for Human Sciences, part of the 
Defence and Evaluation Research Agency. The system consists of an object model database which can be interacted with 
via simulated MAS surgical tools. It is based on two low cost networked personal computers and linked to a pair of 
laparoscopic tools to provide accurate force feedback within a virtual environment. Development of an integrated 
training system forms a necessary part of delivering an effective training tool. A hierarchical task analysis (HTA) has 
been used to determine the key skills demanded of the surgeon in laparoscopic ectopic pregnancy. The experimental 
evaluation of system features to enable cost benefit trade offs to be made is discussed. From the HTA and a review of 
the literature new conclusions about the fundamental nature of the tasks to be trained in MAS are presented. The 
conclusion that adaptation to a continually varying control law is the fundamental task to be trained in MAS has 
implications for the design of MAS training systems. 

A future programme of experimental trials work on the simulator design parameters of force feedback, scene detail and 
3-D, and the application of Distributed Interactive Simulation (DIS) networks, being carried out in conjunction with the 
Surrey RCS MATTU unit and Loughborough University is discussed. 

INTRODUCTION 

The DERA Centre for Human Sciences has a background in developing and evaluating simulation for military sponsors 
and has recently developed a training system for Minimal Access Surgery (MAS). The system VISTA (Virtual 
Interactive Surgery Training Aid) has been described in previous papers (Kelly & Beagley, 1997) and consists of a novel 
hardware platform which includes a deformable database, viewable in 3-D, which can be interacted with via simulated 
MAS surgery tools which have simulated low force feedback (Figure 1). Over the course of the VISTA systems 
development a small number of similar systems have been developed by universities, medical research laboratories and 
industry, with many common features. However a number of issues remain to be addressed by this (and other) systems 
before they are fully taken up by the medical world, and these are discussed below. 



Figure 1. 


* Currently at: Ministry of Defence, SC (LAND)3A, Room 3358 - Main Building, Whitehall, London SW1A 
2HB, U.K 


Paper presented at the RTO HFM Workshop on “The Capability of Virtual Reality to Meet Military 
Requirements", held in Orlando, USA, 5-9 December 1997, and published in RTO MP-54. 
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Training System Requirements 

In an ideal world one should first have an application in mind and then design a system to meet the application needs, 
based on an analysis of the applications requirements, with the design features optimised for this, or a limited set, of 
applications. Systems designed to be all things to all men are rarely cost effective solutions. However techno ogica 
advancement does not usually proceed like this. Most virtual surgical environments have moved from a technological 
concept demonstrator to retrospectively define an application. The best systems have a narrow application focus (e.g. 
training surgical procedures) the less satisfactory systems offer a wide range of applications (e.g. surgical planning 
surgical rehearsal, skill training, and operative support). The value of developing the system design based on a thorough 
analysis of the application, and of selecting design features to provide maximum cost benefit for the chosen application is 
demonstrated by the approach taken for the design of VISTA. 


Task Analysis 

The application focus for the VISTA system is training MAS skills, using a cost effective training system. To identify the 
requirements for the training material an analysis of MAS skills was carried out (Shepherd, 1989) using Hierarchical 
Task Analysis (HTA). This focused on the surgical procedures relating to the condition of ectopic pregnancy. This is a 
relatively common condition which is potentially life threatening (Graber, Allen, Levy, & Talley, 1994). Treatment is 
predominantly surgical and is an area where MAS has made an important impact. The analysis was constructed through 
interview with surgeons. One of the resulting diagrams is presented in Figure 2. 

The trainin'* tasks represented within VISTA include synthetic representations of the basic skills batteries that are 
employed by a number of MAS training hospitals (e.g., Rosser, 1995). However the HTA, along with observations and 
discussions with surgeons has enabled us to identify an additional basic skill component of MAS that is not currently part 
of the skills training in many hospitals and which has important design implications for any simulator that may be used to 

train these skills. 



Figure 2. Part of HTA Decomposition. 
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Initial consideration of the unusual working environment of the minimal access surgeon suggests that they must be able 
to rapidly model and adapt to a variable control law. That is the relationship between their hand movements and the 
displayed control (Laparoscopic surgical tools) movements can continually change from moment to moment as the 
camera’s position, orientation and FOV is changed. This may be the most fundamental skill difference from conventional 
open surgery techniques. The surgeon is continuously sampling the environment, modelling the CD relationship and 
confirming the model by test movements. An example of such a test movement can be seen in the lower level HTA task 
(plan 2.1.2.6.3.3; task 1) “Wipe over with burner to obtain practise run”. Therefore it is a design requirement of a basic 
MAS skills trainer that the camera position can be controlled and varied during a training task. This feature cannot be 
found on some skills trainers which arguably limits their training value. Pilot studies on the rates of learning MAS skills 
in a virtual environment with and without variable camera position have been carried out using the VISTA system 
(Raynor, 1997). These confirm that the skills learnt in a variable camera position regime transfer better to a criteria task, 
than when learnt with a fixed camera regime, and that when MAS skills were leamt in a fixed camera regime interference 
was found on transferring to a criteria task. 

Design Trade Offs 

Developing a cost effective solution was also supported by taking a modular approach to the technology. All elements of 
the development system were hosted separately to enable their contribution to the overall systems training effectiveness 
to be assessed and optimised. The final design solution could then be packaged as a single platform solution using 
hardware and software solutions that would be state of the art at the time of delivery. The contribution of features to the 
overall training effectiveness of the system was the basis of the supporting experimental programme. Systems exist 
which enable force or tactile feedback to be implemented in a virtual surgical simulator (Immersion, 1997), however it is 
the sophistication of the underlying mathematical model of tissue deformation which makes this task easy or difficult to 
simulate. Accurate implementations of force feedback are a high cost driver and may offer low benefits at the basic level 
of MAS skill acquisition. The main areas to be evaluated for their contribution to the training systems effectiveness are 
given in Figure 3. 

FUTURE RESEARCH ISSUES 

The main research issues we are concerned with can be grouped into three areas; the hardware issues, the ergonomic 
issues and the training media and method issues. 


Feature 

Cost 

Technical difficulty 

Training benefit 

3-D 

Medium 

Low 

Unknown/Low? 

Force Feedback, 

Simple Algorithms 

Medium 

Medium 

Low? 

Force Feedback Advanced 
Algorithms 

High 

High 

Medium? 

Polygonal Organ Models 

Medium 

Low 

Medium 

Voxal Organ models 

High 

High 

Low? 

Performance measurement 

Medium 

Low 

High 

Skill based tasks 

Low 

Low 

High 

Variable camera Position 

Medium 

Low 

High 

Olfaction 

Medium 

Medium 

Low? 


'?' indicates areas for further research 

Figure 3. Simulator Features for Design Trade Offs 


Hardware Research Issues 

There are two main areas in the development of the hardware on which SME’s appear to differ; these are the requirement 
for 3-D and the requirement for force and kinaesthetic feedback. This difference of opinion may be because of some 
difference in the applications which the SME are concerned with, or differences in the training and execution of the 
surgeons skills (and the cues they are accustomed to using). Or it may be confusion or unfamiliarity with the potential of 




3-4 


each of these technical possibilities as opposed to their limitations. It is clear that both 3-D and Force feedback are issues 
requiring further research. 

Force/Kinaesthetic Feedback 

Many surgeons say that any MAS training system must have force and kinaesthetic feedback for the surgical tools. But it 
is clear that one of the problems in this environment is the low levels of force feedback present which can be exacerbated 
by the flexibility, damping and adsorption of the surgical tool being used. 

The high cost of force feedback in the system meant that its contribution to training effectiveness was one of the first 
areas to be addressed. Pilot studies (Kelly & Cotter 1998) have indicated that the simple model for force feedback 
implemented in VISTA can have a beneficial effect on the accuracy and speed with which a training task is performed in 
the system. 


3-D 


Early MAS systems did not have colour displays and few now have 3-D. Technology is now able to offer 3-D to the 
MAS surgeon and it is likely that unless there is a strong reaction to it 3-D displays will form part of future MAS 
surgical suits. Nonetheless the benefit from its use is still unclear, and while lightweight headsets that still allow a degree 
of peripheral and local normal vision are now available, some surgeons still find their use uncomfortable and restricting. 
It is clear that many MAS tasks require depth perception but it is also clear that depth cues are present in normal displays 
(e.g. shading, shadows, occlusion etc.) and that not all surgeons benefit from enhanced 3-D. There is some evidence that 
two-handed tasks can be carried out faster using 3-D, but the conclusive research remains to be done. VISTA has a 
selectable 3-D output to enable the contribution of the 3-D displays to training effectiveness to be researched. 

Ergonomics 

The variety of designs of MAS systems available for surgeons indicate that there is still a need for some basic ergonomic 
studies on relative screen, body, and hand positioning, and to consider the possibility in some cases of seating or 
supporting the surgeon. Studies of operating theatre team interactions are being carried out (Helmreich & Schaefer, 1994) 
and VISTA has been sourced to provide the operational task in these studies. 

Training Media 

The development of Objective Performance Measures (timings, error rates, positioning and placing accuracy - see, 
Johnston, Bhoru, Satava, McGovern, Fletcher, Rangel, & Loftin, 1995 ) to support the traditional subjective assessment 
of performance normally found in surgical training is seen as a key area of research for MAS. Objective Performance 
Measures (OPM) will be developed using the analysis of MAS skills and additional information from SME interviews 
and the examination of current training practices. These measures support the setting of training performance standards 
and aid in the provision of feedback during skill development. It is also seen as important that the training material is 
based on the training analysis and on existing training theory. This is to ensure that training is effectively structured, with 
appropriate sequencing, level of exercise detail and feedback. This will lead to the development of a series of training 
tests that can aid in the development of an auditable training trail for MAS. 

CONCLUSION 

It is clear that the new surgical procedures of MAS offer many advantages for patient care and that technology offers a 
degree of solution to the training problem. 

Computer based simulation provides an opportunity to extend the current approaches to MAS training. Simulation of the 
surgical environment combines complete control over the training task with the means of gathering objective 
performance data. 

The rise in litigation in this area has led to demands by surgical training organisations for an accredited, structured and 
validated training programme for MAS, based on Objective Performance Measurement, that can provide a defensible 
audit trail for MAS competence. The degree to which technology provides a solution to the training problem and the 
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exact benefit of all or each of the technical possibilities in this new field is the focus for further research in this 
programme. This paper has presented the initial work towards the integration of an interactive virtual environment with a 
theory based training structure, grounded in the analysis of MAS procedures. In doing so, it has raised some fundamental 
issues relating the skill elements that underpin MAS. 

We believe that a skills trainer that is cost effective can be built, but that a system for full surgical simulation whilst 
possible technically is still too costly for the basic surgical training market place. It is likely that any surgical training 
centre would eventually have a range of training systems each with differing amounts of technical sophistication, 
catering to basic training, advance skill training and full surgical simulation. 
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This presentation describes the operational requirements and methods of achieving remote operation of Combat 
Engineer Equipments for use by the UK Army during periods immediately prior to combat, possibly during combat and 
extensively in battle area clearance operations. The techniques can also be used in peace support. 

The paper examines the teleoperational requirements for the adaptation of existing Combat Engineer Vehicles such as the 
Chieftain Armoured Vehicle Royal Engineer (CH AVRE) also known as AVRE tank, and the Combat Engineer Tank 
(CET) and future requirements for service replacement vehicles such as Future Engineer Tank (FET) and Terrier 
(replacement CET). 

The UK sponsor for this work is the Ministry of Defence, Main Building, Whitehall, London. I would like to thank 
Lieutenant Colonel Philip Poole, SOI Engineer, DDOR (Engr&NBC) for assistance in offering advice and material on 
User requirements. 

DERA Chertsey LS4 department is currently conducting applied research in support of the UK MOD programme for 
Combat Engineer Equipments and Robotic Land Vehicles. 

INTRODUCTION 

Unmanned ground vehicles have applications on the battlefield, for example in reconnaissance and mine clearance, 
where their use reduces danger to men. In this presentation, teleoperation and the technologies involved are discussed 
using examples of successful demonstrations. Current technology can already enable manned vehicles to be converted to 
remote operation for tasks where men would be particularly endangered. The introduction of unmanned ground vehicles 
(UGV) technology should therefore be evolutionary, with the aim of developing vehicles which have greater intelligence, 
independence and versatility, and which could save manpower at favourable costs. 

The idea of robots on the battlefield will conjure up, to many, visions of large-walking creatures roaming free and 
destroying everything they encounter. Whilst the battlefield use of robots of such sophistication is still a long way off, 
UGV’s of more limited capability are already making an important contribution to the modem army: UGVs are now 
widely used for explosive ordnance disposal (EOD) tasks, and have been used in this role for over 20 years. 

Some advantages of using UGVs instead of manned vehicles are: 

• Reduced risk to man (since destruction of an unmanned vehicle does not result in the loss of the driver and 
crew). 

• The possibility of bolder concepts of operation, because of this reduced risk. Higher performance levels on 
extended and repetitive operations (where, for example, humans may suffer from fatigue, boredom or stress). 

• Manpower reduction (as robotics compensate for a reduction in manpower without loss of effectiveness). 

Conversely, disadvantages of the vehicles are: 

• Their high cost. 

• A lack of user confidence, acceptance and an increased logistic burden. UGVs are probably less flexible and 
adaptable than men. 
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User Requirements 

Over the years, the need for, and scope of, Engineer support on the battlefield has been well established. It is provided by 
a wide variety of equipment and vehicles, optimised for their specific role and place on the battlefield. Engineers have, 
and will continue to use, a number of armoured vehicles to provide mobility, counter mobility and survivability support 
to formations whilst providing protection to their own crews. Combat Engineer Tractor (CET), Engineer tanks, the AFV 
432 and CVR(T) all fulfil vital functions within Engineer regiments. It is planned to replace all of these vehicles in the 
next 10 years. 

The current Chieftain Armoured Vehicle Launched Bridge (CHAVLB) and Chieftan Armoured Vehicle Royal Engineer 
(CHAVRE) were converted from Chieftain gun tanks in the 1970s and 1980s respectively, to replace their Centurion 
based predecessors. Experience gained on operations and in training and Operational Analysis have highlighted a major 
deficiency in the mobility of Chieftain Engineer variants when compared with the more agile armour that they support. 
This deficiency was exemplified on Op GRANBY (Gulf War) when Engineer vehicles fell behind between 3 and 6km, 
for every 10km advanced. The replacement of both vehicles will be carried out with Future Engineer Tanks (FET) having 
an in-service date of 2002/3 and the replacement of CET with the Terrier vehicle in 2006/7 will enable Engineers to 
provide improved close mobility support to armoured formations. 

The mechanisms of Engineer Support, Engineer tasks on the battlefield involve mobility, counter mobility and 
survivability support. Tasks are undertaken throughout the width and depth of the area of operations, and are provided by 
a range of equipments with complementary characteristics and capabilities. Engineer mobility support tasks include wet 
and dry gap crossing, obstacle breaching, route opening, route maintenance, countermine and EOD battlefield clearance. 
Mobility is denied to an enemy through the enhancement of natural obstacles, the construction of man-made obstacles 
and the blocking of routes. The ability rapidly to construct earthworks and defences for all-arms protection is an essential 
part of survivability support. 

It is the aim that the Robotic Land Vehicles research programme will help to enhance the Royal Engineers and other 
arms capability by improving mobility and counter mobility support to armoured formations. 

Direction of UK RLV Programme 

Two of the equipment types are discussed here where DERA have to date conducted research into teleoperation for 
remote driving and engineer tasks. Other applications for other Engineer vehicles will be likely in the future. The need 
for teleoperation techniques will most likely be in colour and contain elements of mono/stereo vision, augmented reality, 
virtual reality (VR) or a combination. 

The programme has so far been concentrating on teleoperation kits for in-service vehicles such as the CHAVRE and the 
CET. It is seen as the most likely route for robotics to enter service and gain user acceptance. Acceptance of new 
technology equipments has always been a challenge to the researcher and the growing requirement these days is to show 
that the benefits to the user outweighs the cost of development. 

We are currently investigating technologies that will improve teleoperation capabilities - these are stereo vision, optical 
flow, communications, image compression and augmented reality techniques. 

Design Philosophy 

The philosophy for the core area of the DERA research programme has been to take the latest available technology and 
assess its capabilities for the users requirement. Urgent operational requirements (UOR) occur from time to time. By 
assessing the latest technologies/equipments, rapid responses can be made when the need arises to provide applique kits 
for UORs. 

The use of such kits allows flexibility where the kit can be removed reasonably quickly from the vehicle and it can then 
be interfaced with another type of vehicle. Commercial off the shelf equipment is used wherever possible. There must be 
no interference with the manual operation of the vehicle and a short change over time from manual to remote control by, 
for example switched operation. Design concepts must be space conscious as the controls and interface units must fit into 
already cramped spaces in the Engineer vehicles. Operator Control Units are required to be fitted into the secondary 
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vehicle. Space is an important consideration and the use of Helmet Mounted Displays or VR systems may help to 
alleviate this problem. 

A recent injection of finance into the research budget has allowed the direction of the programme to investigate future 
teleoperation techniques such as novel vision and augmented reality for teleoperation. 


Teleoperation Kit For Chieftain AVRE 

Applique kits for possible UOR use in Bosnia and also as proof of principle for future FET and Terrier programmes were 
investigated in 1995/96. The system currently being used in the US SARGE programme was purchased and minor 
modifications were made to suit the UK MOD requirement. The AVRE mineplough and fascine operations were tested, 
using monovision teleoperation to drive the vehicle remotely and carry out the manipulative tasks. Other areas for 
consideration for teleoperation are, use by a single operator, selection of cameras and camera positions, the use of a 
microwave link for a line of sight video transmission from AVRE to the remote command vehicle. 

Three cameras are fitted to the front of the vehicle and one at the rear. The centre front is mounted above the driver’s 
head which improves forward vision by reducing obscuration by the ploughs. Two side cameras improve side/peripheral 
vision and allow accurate driving within narrow lanes or fenced off areas such as minefields. The rear camera allows 
accurate and fast reverse driving. Operator control units can be fitted to display single or multi image options. 

Teleoperation Kit For Combat Engineer Tractor 

Teleoperation for CET has been based on the methodology and hardware used on AVRE. From the AVRE trials, driving 
functions were not difficult, thus the CET research has been aimed at investigating and improving the manipulative tasks. 
Trials were conducted in spring 97. The main technical objective was to investigate digging tasks required in earth 
moving, the construction of obstacles such as sand banks and tank ditches. Clearance of obstacles was investigated, for 
example, filling in NATO standard tank ditches. Other clearance activities require the removal of heavy obstacles. For 
this, the CET 4-in-l bucket is ideal for these manipulative type tasks, but not easy to implement because of limited visual 
perception. To carry out these tasks, camera positions and the numbers of cameras must be considered. 

Operator Control Unit And Its Functions 

The OCU unit was selected for its history of performance use on the US SARGE programme. In the UK research 
programme, it is used to perform driving and manipulative tasks. It can be purchased from Omnitech Robotics or Robotic 
Systems Technology and consists of a 486 33MHz PC with video LCD display and connections for interface outputs. 
Driving is via a motorbike handlebar control offering various functions such as steering, accelerator, braking and gear 
change. Control of the ploughing and fascine operations is done with the joystick. 

Other functions have been modified to provide the 4-in-l bucket controls. The bucket can be operated by joystick or by 
standard bulldozer lever type controls to provide such functions as raise/lower, digging/dumping, open/close and 
float/level grading. 

The PC is able to offer many diagnostic capabilities - for our purposes items such as, gear status, vehicle speed, low oil 
pressure warning, safety status warnings, communication link dropout etc. have been incorporated. 

Communications And Safety 

Communications for remote links from the command vehicle to the remote vehicle offer control of the remote vehicle by 
two way radio link. The military vehicle radio, in the case of AVRE/CET is a VHF Clansman and was used to provide 
command functions, (e.g. engine start/stop, steering, accelerator and braking). Operation is at 4.8k baud rate and 
modulated into voice format. To prevent communication problems, two switchable antennas are used, one positioned 
90degress out of phase with the other antenna. This method of control is suitable with military radio links such as the 
UK’s next generation Bowman system. 

The tele-link is only required in one direction from the remote vehicle to the command vehicle to be able to carry out the 
visual driving and manipulative tasks. We have used a 1.3GHz FM commercial microwave link (Gigawave Antennas 
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UK) with circular polarised antenna for all round performance. The main problem is, this method generally requires line 
of sight communication to transmit the high bandwidth video image. Alternatively fibre optic links that offer high 
bandwidth can be used but the trailing of cables over l-2km can be a problem in some scenarios. 

Safety is paramount when remotely controlling the vehicles. The computer hardware has a safety unit with 
watchdog/command data safety flags. An in vehicle safety cut-off switch is also incorporated. This feature is essential 
when conducting evaluation trials, some of which do take place at the researcher’s site. Some vehicles lose their 
hydraulic breaking when the engine is switched-off, an important point to remember when considering safety. Software 
is required to be fail-safe but not safety critical. 

Teleoperation Proving Trials 

Trials were conducted in the UK to assess remote driving(1996) and engineer vehicle tasks (1997). Using colour 
monovision equipment, the MMI trials proved that driving was relatively easy but most other tasks varied in degrees of 
difficulty. Breaching ditches, grabbing and lifting was successfully achieved. With the 4-in-l bucket, monovision 
highlighted perception problems during digging, such as loss of distance perception, depth and slope angle accuracy. The 
engineers became reasonably proficient within one day after using the equipment. Remote operations generally took up 
to 25% longer than when using the equipment manually. 

ONGOING AND FUTURE WORK 

Future topic to be covered are: 

• Modelling for optimisation of operator perception and awareness, trials of stereo vision kit on the CET, 
assessment of head mounted displays and LCDs for stereo vision, use of inclinometers to judge slopes, the use of 
force feedback to “feel” driving and manipulation of the bucket, night driving - use of night vision equipment. 

• Provision of increased user awareness, co-ordination with other nations and groups - US, France, SILVER (UK 
-DERA/Industry/Academia) and also keeping a watching brief on future potential technologies. 

The core activity of our research for DDOR (Eng&NBC) continues to be improvements to in-service vehicles through 
current and future technologies such as VR. It is likely therefore that areas discussed in this workshop will feature in the 
future of remote control of Robotic Land Vehicles. 

CONCLUDING NOTE 

I believe that UGVs will become increasingly important for RLV tasks as they can be used on hazardous tasks without 
endangering men. 

Reduced levels of manpower in the modem army need to be compensated for by increased levels of technology if the 
same or greater level of operational effectiveness is to be achieved. Repetitive and manpower-intensive tasks, such as 
convoying, logistic re-supply and security patrolling, could be carried out reliably by remote equipments and for longer 
periods, and would thus free people for other demanding tasks. 

Bearing recent events from around the world in mind, it seems increasingly likely that the army of tomorrow will find 
itself involved more and more in operations other than war. Peacekeeping operations, with the intrinsic political 
unacceptability of loss of life, seem sure to promote the case for using teleoperated UGVs. 
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USING THE VIRTUAL REALITY MARKUP LANGUAGE (VRML) TO 
UNDERSTAND THE U.S. ARMY’S SIMULATION AND TRAINING DOMAIN 

LTC George F. Stone III, Ph.D. 

Center for Land Warfare 
200 Army 

The Pentagon (1D536) 

Washington, DC 20310-0200, USA 

Knowledge gained via the technology explosion continues to grow at an exponential rate. The size of all knowledge 
bases will become larger and more difficult to design. The Functional Description of the Battlespace (FDB) is the Army 
domain modeled for use in simulation development, Warfighter’s Simulation 2000. The database that must be designed 
to describe everything that the Army does is going to be very large. Military unit descriptions, mission models, task 
process descriptions and other models reside within a large warehouse of data, models, algorithms, information and 
knowledge. 

FDB users currently navigate on two-dimensional screens that use standard hypertext markup languages. In ordei to lead 
the FDB user in a more efficient and effective manner, a three-dimensional mapping and visualization of the FDB 
contents are sought The gain will be increased understanding of the FDB knowledge warehouse. Using the virtual 
reality markup language (VRML), a virtual map and guide of the FDB will teach the abstract concepts inherent in 

complex domain modeling. 

The FDB is becoming the Army’s single source repository of validated, verified and accredited descriptions and 
performance parameters for all entities in the battle space. The data that populates the FDB spans the following 
categories and their interactions: 

1 Human Characteristics: Performance data of behavior, capabilities and limitations, learning, thinking, situation 
awareness, judgment and the influences of doctrine, organizations and the physical environment. Mission models and 
task process descriptions are included in this category. 

2. Systems and Materiel: Descriptions and performance data and utility data for individual and crew served 
battlespace systems as influenced by human operators, the natural and manmade physical environment, and impact of 
supply and maintenance operations. Equipment models show this type of information and data. 

3. Physical Environment: Descriptions of natural and manmade physical and environmental conditions and 
effects including atmospheric, terrain, oceanographic, and space components for all climates world wide, including 
dynamic changes to the selected scenario battlespace that would occur due to natural environmental disasters (i.e. floods, 
earthquakes, sunspots, destruction of natural or manmade objects such as lakes, dams, etc.). 

4. Organizations: Arrangements (templates) of systems into units following prescribed command structures to 
perform roles and execute mission objectives. This will represent the impact of cognitive processes of command and 
control, standard operating procedures (SOP), status of communication links with higher, adjacent and lower echelon 
units and the influences of outside activities on mission completion. Unit models contain this information. 


Paper presented at the RTO HFM Workshop on “The Capability of Virtual Reality to Meet Military 
Requirements ”, held in Orlando, USA, 5-9 December 1997, and published in RTO MP-54. 
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TRANSFERRING VR TECHNOLOGY BETWEEN MILITARY AND NON¬ 
MILITARY APPLICATIONS 


Thomas W. Mastaglio, Ph.D. 

Old Dominion University 
Virginia Modeling, Analysis & Simulation Center 
7000 College Dr. 

Norfolk, VA 23435, USA 


ABSTRACT 

Virtual Reality (VR) technology for commercial applications has been oversold to the public during the past 2 to 3 years. 
There have, however, been relatively few success stories outside of entertainment, but that does not mean VR cannot 
meet the requirements of non-military applications. It is important that we take a broader view of VR than helmet- 
mounted displays and wearable technology, one that includes interfaces able to display high fidelity 3D environments 
and models In this paper I discuss the methodologies for accomplishing technology transfer. These include cooperative 
agreements, corporate funded R&D, and government funded initiatives. The requirements of commercial application 
domains, including industrial operations, government (non defense) operations, healthcare and entertainment are 
discussed and related to military training systems. Last I will propose ways that these two communities can work 
together on joint research and development, transferring technology bi-directionally, and the use of forums to enhance 
dialog and discussion. This paper is intended to stimulate a discussion of issues (such as intellectual property) which are 
appropriate agenda items for future joint discussions and workshops supported by both defense and non-defense VR 
developers. 

INTRODUCTION 

Virtual Reality (VR) captured the public’s imagination and has been the darling of the technical communities that 
develop cutting edge computer interfaces and graphics computing capabilities. But like so many other promising 
technologies, it has been slow to mature sufficiently to be affordable, reliable, and to possess sufficient capability to meet 
the incited expectations. So now we hear plaints that VR is not really needed, that is too expensive, too slow, and too 
hard for its users to make a conceptual leap from what they see to the features of the real world a particular VR-based 
synthetic environments allegedly represented. 

A different perspective is needed in order for VR to realize its potential. First, I propose we begin from an application 
perspective rather than, the more frequently taken, presentation viewpoint. Rather than being enamored with the “gee 
whiz” nature of VR, we need to see it as a feature of simulation technology, specifically user interface methodology. 
Furthermore, we should think of simulations as tools that aid their users in making decisions. These decisions can be for 
the purpose of learning, managing, or even of being entertained. But in any case the users have a certain goal in mind 
when they use a computer-based simulation. The user’s interface to the simulation should be chosen or designed with a 
notion of that goal in mind. 

Simulations are intended to be a model of some phenomena which exists in the world. Those phenomena could be some 
part or aspect of the world that is directly observable as well as one that is subatomic or even imaginary. In either case 
the use of VR as an interface mechanism exists solely to aid users in mapping the representation in computer simulation 
to their own conceptual understanding. With these perspectives in mind, let us consider how the needs for adequate VR 
technology to meet the needs of both military training and the commercial world complement one another and what we 
can do to collaboratively develop those capabilities. 

METHODOLOGIES FOR ACCOMPLISHING TECHNOLOGY TRANSFER 

Technology that has potential for dual use in both military and commercial settings has been long sought after. In the 
United States the federal government has tried mandating these efforts, encouraging government agencies to seek out 
opportunities to transfer their good ideas to commercial practice and even directly providing funding to accomplish 
technology transfer. While the notion has obvious appeal it has most often been unsuccessful because of the difficulty in 
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stimulating a market for what are most often technologies for producing or defeating violence. Computer technology, 
especially better approaches to interfacing users to software, is a technology that can serve multiple purposes right from 
its development. We will examine some approaches to accomplishing dual use of VR technology. 

Cooperative agreements between government development activities, such as laboratories or procurement commands, 
and non-defense activities, both commercial companies and university-sponsored research centers, should be considered 
as a way to reduce development costs. Cooperative agreements can be used to either leverage available funding or 
reduce costs of development by using the resources which a university research center offers. The difficulty with these 
agreements is insuring that property right issues are clearly addressed in the agreement itself so that future disputes over 
ownership and the rights to apply the results are clearly defined. I recommend that no one be permitted exclusive 
ownership of any cooperatively developed technology or product, rather those should be shared with full access to either 
party. 

It might appear that corporate funded research and development will not achieve the goal of providing needed 
technology to the military. However, often contractors and companies, especially defense contractors, expend their 
limited R&D funds developing capabilities that do not meet the requirements of the military. Sometimes these 
capabilities have commercial application, but dual use technologies able to meet both defense requirements and find a 
niche in the marketplace are the ideal. The way for government activities to best leverage corporate R&D is to willingly 
define their requirements to industry. Dialog between government and commercial industry are important, there should 
be open exchange of information at the technology level. This does not mean that the possibility that something new, 
innovative, and revolutionary being developed by industry should be overlooked. There will always be innovations that 
will serve new purposes that are not yet defined. Government agents need to be alert to these, especially those that will 
improve defense training or combat capabilities enhancing combat effectiveness. 

Last, there will always be a need for government funded research initiatives. Some government funding is needed to 
cultivate new technologies that will improve combat effectiveness because warriors can train better or operate more 
efficiently. Government scientists should not rely on their own laboratories to innovate all new technologies for their 
militaries. Defining what development needs to take place and sponsoring funding for it outside of government 
laboratories is still critical. A suitable strategy for meeting the technology needs of future military forces requires R&D 
organizations to adopt a strategy that includes all three types of approaches to leveraging capabilities outside of their own 
organizations. Cooperative efforts, dialog with industry on needs to guide corporate R&D investments, and government 
funded research should be components of such a strategy. 

REQUIREMENTS FOR COMMERCIAL APPLICATIONS DOMAINS 

I will discuss some of the basic requirements for VR interfaces to simulations in key non-military application areas. 
Keep in mind the perspective proposed in the introduction: that these simulations are tools that support decision-making 
and may benefit from the ability of VR technology to provide an immersive interface to that simulation. The areas for 
discussion are defined in four broad areas: industrial operations, government operations, healthcare, and entertainment. 

Simulations to support industrial operations are not necessarily new, operations researchers, management scientists, and 
industrial engineers have used simulations to support their analyses for years. What is new is the opportunity to use 
simulations at a higher conceptual level to build and run on desktop platforms, enterprise level models. These 
simulations capture the emergent properties top-down for a given business enterprise. Today we can model entire 
factories, distribution systems, processing plants and etc., delivering those simulations to the manager’s computer 
desktop for him or her to use for strategic planning and day to day operations. Given that these types of simulations will 
serve not just analysts but managers as well, we must consider when immersive technology adds to the ability of the non¬ 
technical user to relate what the computer presents to his or her understanding of the real world features being 
represented. VR technology will be of use if it is affordable and easy to use. Strap-on wearable equipment will probably 
not find favor with management level decision-makers. However, high resolution three dimensional presentations 
designed to aid understanding will find an eager market. 

Local and state governments must meet an ever-increasing number of informed constituent groups. They are being held 
accountable for social programs, economic development and efficiency of public services. Simulations which, like those 
discussed above for industry managers, capture the overall characteristics of a government operation and which can be 
used to better plan for the future and manage day to day operations will be soon available to government officials. These 
simulations need an interface that not only helps decision makers, such as city managers and their staffs, do their jobs 
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better but are also suitable for explaining decisions to elected officials and the electorate. Planned investment in new 
infrastructures, such as schools, emergency services, or transportation, will be simulated to make the best choices and 
then demonstrated to the public. Three-dimensional high-resolution display technology as part of a simulation system 
based on GIS or other existing terrain databases will be in demand. These are the same features military trainers and 
operational planners want in their training or planning systems. 

Healthcare needs simulations to help them more efficiently manage their operations and train medical personnel. Patient 
simulators are already under development. These full-size replicas of humans can train healthcare professionals in 
proper procedures for handling emergency medical situations. A VR system that permits personnel to practice similar 
procedures would also be widely accepted. This is a case where both the ability to display three dimensional images, but 
from a more limited image database than needed in the above domains, is important, and where the immersive 
capabilities of the system are important. The requirements for VR technology able to support similar medical training in 
both the commercial and military communities exists, but can serve other military training needs as well (e.g., 
dismounted warfare training). Also, managers of facilities and programs in healthcare will, in the near future, be seeking 
simulations to help them better manage their enterprises. Their requirements for VR technology will be similar to those 
of industrial and urban operations discussed above. 

Entertainment has been viewed as a natural ally and collaborator in developing VR technology. The military training and 
entertainment communities have common interface capabilities per se, HMDs, interaction devices, and display screens. 
Collaborative efforts are therefore worth pursuing. One must understand though that entertainment requirements and the 
liberties taken in how they use the technology may impact the ability of what is developed to meet military requirements. 
Entertainment does not have to concern itself with fidelity to real world constraints (i.e., the laws of physics). It is more 
concerned with establishing a story telling context. Also entertainment can often afford to sacrifice scene fidelity for 
cost. Military training systems have to insure that whatever is delivered into the hands of warriors matches fidelity with 
tasks being trained. Sometimes the goals of entertainment and training are in opposition. I recommend that our best 
approach with entertainment is to let those organizations develop technology based on their profit incentives and 
determine, after it is completed, what capabilities it has to meet military requirements with or without modifications. 

COLLABORATION BETWEEN DEFENSE AND COMMERCIAL APPLICATIONS 

Given that there are common needs and opportunities to collaboratively develop technology between defense and some 
commercial areas I would like to discuss some ways to implement that collaboration. 

Joint research and development involving government agencies and either academia or industry is probably the most 
difficult approach. It requires a clearly defined objective and plan. The agencies involved must agree early on what the 
project expects to accomplish. Each party involved should insure that those objectives will help them to meet their own 
organization’s requirements when completed. This is key to embarking on a joint project, but more important to the 
overall success is a commonly agreed upon project plan with a specific schedule that includes milestones, planned 
reviews, and an agreed upon commitment of resources. R&D often has poorly defined end points and in many cases that 
is not necessarily bad, but in the case of joint efforts, better definition is required to insure the parties involved do not feel 
that the project failed them. 

Frequently overlooked is the opportunity to transfer technology bi-directionally. Not only can government developed 
technology frequently be applied to meet commercial needs, but existing commercial technology is often able to meet 
military requirements with little or no modification. A model for accomplishing this is to establish partnerships between 
government agencies and other research centers that allow the research center access to government technology. This is 
usually accomplished through some sort of cooperative research agreement or memorandum of understanding. The 
partnership also should include a complementary contract vehicle that allows the government activity to procure honest 
broker services from the research center to help them evaluate commercial technology. 

Information exchange between commercial and military sectors is probably the easiest way for these two communities to 
exchange useful ideas. Journals and conference proceedings that describe research are valuable. However, workshops 
and the actual conferences themselves are perhaps best. These types of forums allow researchers to dialog about what 
has been accomplished and understand best how to potentially apply it to meet their needs. They also allow ideas and 
requirements for future research to be exchanged. 
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SUMMARY 

VR may not have met the public’s expectations based on its hype as presented in the media but we, as a community, 
should not become discouraged. We must continue to work not just to improve display technology, a lot of which is 
being done by the OEM community to meet other needs (such as entertainment), but to work collaboratively with two 
purposes in mind. First we need to understand where common technical requirements exist that can help each meet our 
own requirements. Second we must be able to recognize and develop new capabilities in joint projects or by information 
exchange early in the R&D lifecycle. Only through joint and cooperative research and information sharing will VR 
technology move ahead and eventually meets the promises everyone holds for it. As a community we need to commit to 
working more closely together and enter into a dialog that addresses the issues involved in such collaboration. 



HUMAN FACTORS ISSUES IN THE APPLICATION OF VIRTUAL 

REALITY TECHNOLOGY 

Day Chairman: Robert Breaux 


NAWC-TSD 
2350 Research Parkway 
Orlando, FL 32826-3276, USA 

NOTES ON DAY 2 OF THE RSG28 WORKSHOP 


INTRODUCTION 

The subject of Day 2 of the December 1997 Workshop of the RSG 28 was Human Factors Issues. This subject formed a 
logical transition from Military Requirements of Day 1 and Assessment Methods and Military Applications of Day 3. 

As an introduction to Day 2 the chairman suggested the following issues: 

• Training & Information Systems 

—Cyber Sickness 
—Taxonomy of Proper Uses 

—Measures of Performance / Effectiveness / Outcome 
—Authoring / Design Tools 

• Sense of Presence 

—3D Audio 
—Speech 
—Haptics 
—Olfactory 
—Avatar 
—Visual 
—Locomotion 
—Depth Perception 
—Localization / Manipulation 
—Temperature 
—Humidity 

As described below, the speakers covered many of these issues in their discussions, how VR was applied, which human 
factors were involved, what evaluation results they obtained and what future plans they had. Also, assessment methods 
were presented: what was being assessed and how, strength and weaknesses and potential uses. 
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PROGRAM 


Mr. S. Grant 

(CA) 

Navigation In Virtual Environments 

Dr. J. Templeman 

(US) 

Performance Based Design of a new Virtual 

Locomotion Control 

Dr. R. Fischer 

(US) 

Methods for Improving Depth Perception in HMDs 

Dr. R. Kennedy 

(US) 

Capability of Virtual Environments to Meet Military 
Requirements 

Dr. P. DiZio 

(US) 

Side Effects and Aftereffects of Immersive Virtual 
Environments Created with Wide Field of View, 
Helmet-Mounted Visual Displays 

Dr. N. Beagley 

(UK) 

Human Body Modeling as a Human Factors 

Engineering Tool 

Dr. R. Carrier 

(FR) 

VirtualMan: Fully Articulated Man Model 

Dr. P. Werkhoven 

(NL) 

How Real Are Virtual Environments? 

Dr. B. Knerr 

(US) 

Interface Issues in the Use of Virtual Environments for 
Dismounted Soldier Training 

Mr. S. Davidson 

(US) 

Development of a Virtual Environment Software 

Testbed using COTS Software Components 


In particular, the speakers addressed the following in detail. 

Grant 

Navigation, locomotion, and depth perception in VE 

• Users are greatly disoriented in VEs 

• Walking interface produces better transfer of training than joystick (shorter distance but not time) 

• No difference in simulator sickness between two interfaces 

• Navigation benefits from internal cues 

• Non-visual information well-suited to navigation in areas that appear different from training (smoke, lack of 
visual data, traveling out of sight of rehearsed areas) 

Templemen 

Features of Gaiter 

• Can preserve metrics between physical & virtual space 
Example: Map knee extension into virtual displacement 

• Virtual motion occurs as the knees rock back and forth 
Do not have to wait for the gesture to be completed 

• Gesture-less Gesturing 

No special vocabulary to separate forward, side, & back steps 

• Equipment is compact & relatively inexpensive 
Wireless version can be built today, at greater expense 

Fischer 

• Simulation using HMDs has a lot to offer 
Size, weight, cost, portability 

• Problems are associated with scene immersion, resolution, and depth perception 

• Adding cues of variable convergence and accommodation in synchronism with eye fixation point will help 
significantly as will blurring of far background 

• Efforts will lead to viable implementation of these techniques 





Nl-3 


Kennedy 

Cybersickness in VE 

•VR systems which convey feelings of presence can produce reports of symptoms similar to those of motion and 
simulator sickness 

•Exposure to VR symptoms can also produce aftereffects such as postural imbalance and aiming biases 
•The former problems can influence users to not use the system but the latter can have safety implications 

•Preliminary findings from 3 VE systems show weak or zero correlation between the subjective and objective 
measures of aftereffects which implies both types of disturbances need to be measured in evaluation of VE 
systems 

•The fact that sickness severity does not appear to be correlated with either of the objective measures of exposure 
aftereffects such as past pointing and ataxia may suggest different neural pathways may be involved in these two 
types of aftereffects 


Dizio 

• Motion sickness and postural instability are serious side effects and aftereffects of a 15 min session with a wide 
FOV HMVD 

• Individual susceptibility is very stable 

• Aftereffects involve sensitization: 

- symptoms which appear to be gone in a resting subject reappear in full force - - with minimal exposure to real 
or virtual motion 

• Delays in visual update contingent on head tracking, FOV and resolution are etiological factors 

• Visual update delay is a factor specific to HMVDs 

• No external visual or real motion is necessary 

• Voluntary head movements are provocative 

• Visual update delay somewhere between 20 and 40 ms is the threshold for eliciting motion sickness during a 15 
minute exposure, for wide FOV HMVDs 

- Establishes a target for system specs 

- Provides the rationale for developing incremental exposure routines to prevent side effects 

Beagley 

Body Modeling for VE: addressed the following commercial models 

• Andriod: Mechanical Dynamics 

• COMBIMAN: CERIAC 

• Crew Chief: CERIAC 

• ERGO: Deneb 

• JACK: Transom 

• Kinemation: Alias Wavefront 

• MannequinPRO: HumanCAD 

• MDHMS: Boeing 

• Poser: Fractal Design 

• Safe works: Genicom 

• Sammie: British Technology Group 


Carrier 

Discussed the following issues regarding modeling 

• Definition of grid and key postures 

• 3D Scans 

• Extraction of digitized data from raw data 

• Man model definition 

• Layout of digitized data onto man model 

• Validation 
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Werkhoven 

Localization of objects in VE 

• Spatial perception in VE is not yet as good as in the real world 

• Virtual hand control is promising: 

- fast and accurate 

- likely transfer of acquired visual-motor skills 

• Design tasks: discrepancies because of 

- too coarse modeling 

- inaccurate tracking 

- lack of force feedback 

Knerr 

• VE Performance is worse than Real World performance on a variety of visual tasks 

- Contributing factors are likely to be FOV and resolution 

- Distances in VE are underestimated 

- Stereoscopic displays improve 
performance only at short distances 

- Head coupling sometimes improves 
performance 

- Image fidelity may be a consideration for 
transfer 

• Locomotion simulators can affect the perception of distance in VE, but not always in the ways we would have 
predicted 

• Compensatory cues can affect distance perception in VE 

• What constitutes a “good” locomotion simulator is likely to be task dependent 

Davidson 

Lessons Learned from testbed development 

• Prototyping => VE Authoring Language 

- Describes behaviors, object attributes 

- Event driven/programmable 

• Integrated dB Manager/Multi-process 
harness 

Performance, performance, 
performance 

Open architecture => 3rd party 
attributes/processes 

• Portability 

Binary object database / ASCII 
authoring language 

CHAIRMAN’S FINAL SUMMARY 

Upon conclusion of the speakers and the discussion of their papers, the chairman concluded the day by leading a 
discussion of the need for a technology roadmap. This would promote not only collaboration among research personnel, 
but also demonstrate to our respective nations’ budgeting authority that significant technical progress is being made 
without unnecessary duplication of effort. A prototype roadmap containing technical issues, technical interchange 
points, and a time scale was presented and discussed. 
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NAVIGATION IN A VIRTUAL ENVIRONMENT USING A WALKING 

INTERFACE 

Stuart C. Grant and Lochlan E. Magee 
Defence and Civil Institute of Environmental Medicine 
1133 Sheppard Ave. 

North York, Ontario M3M 3B9, Canada 

For years, aircraft crews and armored fighting vehicle crews have benefited from training in the benign, controlled, and 
instrumented situations created by simulation. Virtual reality (VR) technologies are now starting to bring these benefits 
to the training of dismounted soldiers. 

Simulation for dismounted combatants has numerous applications. In current combined arms simulations, only 
constructive infantry, controlled from a workstation, are available. Their lack of sophistication, relative to their human 
counterparts, is thought to detract from the validity of the combined arms simulation. Adding virtual infantry would 
increase the benefit to all participants. Reconnaissance forces and special operations units could plan and rehearse 
missions using simulation. The ability to explore a future area of operations, such as an airport where hostages are being 
held, from a first person perspective would allow forces to assess and rehearse lines of fire, escape routes, and fields of 
view. Similarly, personnel who must enter toxic environments, such as ship’s damage control parties and nuclear power 
station maintenance crews could rehearse different scenarios ahead of time and familiarize themselves with locations 
they have never visited. 

A HUMAN FACTORS PROBLEM: NAVIGATION 

A human factors problem threatens to prevent dismounted combatants from obtaining the full benefits of simulation. 
People are prone to becoming lost in virtual environments (VEs) (Durlach & Mavor, 1995; Psotka, 1995). This is 
obviously a problem, for being lost impairs performance in the virtual environment and it means that there is little spatial 
knowledge being gained for transfer to the real world. 

Conventional movement interfaces for VR, such as joysticks and 3D mice, likely contribute to disorientation because 
they do not provide the vestibular and kinesthetic feedback people use to navigate in the real world (Loomis, Da Silva, 
Fujita, & Fukusima, 1993; Mittelstaedt & Glasauer, 1991; Schmuckler, 1995). Thus, only visual information is available 
for navigation by dead reckoning (Gallistel, 1990), and that source of information is constrained by the limited vection 
produced by the narrow fields of view obtained in most VR displays. Seeking to address this problem, Iwata and 
Matsuda (1992) provided users with special roller skates to move through a VE. The effect of their interface on users' 
distance perception in a VE relative to a point-and-fly interface was equivocal, however. Slater, Usoh, and Steed (1995) 
used the presence of foot motion to move users in a VE and found positive effects on feelings of immersion, but they did 
not measure navigation effects. 

AN EXPERIMENT 

This experiment examined the role of proprioception in the navigation of VEs by letting subjects explore a computer 
model of a large, complex building, the Ontario Science Centre (OSC), using either a walking interface or a joystick. If 
proprioception facilitates navigation in VEs, subjects using the walking interface should be better oriented in the VE and 
acquire more knowledge of the OSC for transfer to the real world. To aid the interpretation of the results, the experiment 
included control groups that either studied a map of the OSC, walked through the real OSC, or received no information 
about the OSC. 

The VE 

The OSC is an excellent place to study navigation because it has an unconventional floor plan and contains many unusual 
items, thus providing unique and memorable locations while preventing subjects from finding their way using heuristics 
derived from their experience with more common buildings. For this experiment a model of 7,200 square meters of the 
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OSC, including hundreds of exhibits, was constructed using shaded polygons. Eight of the exhibits were selected as 
destinations for the navigation task according to their location in the OSC, their size, and their distinctive appearance. 

The Walking Interface 

The walking interface was a platform 1.22 m in diameter surrounded by a hand rail. The surface of the platform was 
covered with a slippery laminate that allowed subjects to easily slide their feet across the surface. 

Subjects wore a magnetic sensor on each foot. By walking in place, the subject’s foot motion moved the subject s eye 
point in the VE. Horizontal foot movement, either forward or backward along the subject’s body direction produced 
forward movement in the VE. This is similar to normal walking, where the head moves steadily forward as the feet move 
back and forth beneath the body. The farther and faster the subject moved his feet on the platform, the farther and faster 
he moved in the VE. On the basis of a pilot experiment, the magnitude of the foot movement was amplified to produce a 
subjective match between walking speed and visual flow. 

Design of the experiment 


Eighty subjects, all older than 15, completed the experiment. The nature of recruiting resulted in a different ratio of men 
to women in each experimental condition, but because inclusion of gender as a covariate in the analyses lepoited below 
did not change any of the results, the effects of gender will not be reported. 

Five groups of 16 subjects took part in the experiment. The VR-walking group explored the computer model of the OSC 
using^the walking interface. The joystick group did the same using a joystick. The map group explored the OSC using a 
floor plan that contained representations of all exhibits, and by looking at photographs of the destination exhibits. The 
real-walking group explored the real OSC. The no-experience group received no information about the spatial 
arrangement of the OSC, but they did see photographs of the destination exhibits. 

In the first phase of the experiment, subjects in the VR-walking, joystick, and map groups completed a simulator sickness 
assessment. The assessment consisted of a 28 item questionnaire (the Simulator Sickness Questionnaire, the SSQ; see 
Kennedy, Lane, Berbaum, & Lilienthal, 1993) and an eyes-open Romberg balance test, performed on a force platform. 

In the training phase, the experimenter guided all subjects, except the no-experience subjects, through one circuit of the 
OSC, visiting the eight destinations. The VR-walking and joystick groups did this by moving through the computer 
model. The map group was directed along a path on the map, viewing photographs of each destination as they reached it 
on their map. The real-walking group followed the tour in the real OSC, and the no experience group only viewed the 
photographs of the destinations. At the start of the tour the experimenter told the subjects that they would be asked to find 
the destinations later in the experiment, and that they should attempt to learn the location of the destinations. 

To determine how well the subjects were oriented in the VE, each time the VR-walking and joystick groups reached a 
destination during the training phase, they were asked to face back to the origin of their VR tour. If they were disoriented, 
they should make large errors, but if they were not, their errors should be small. To provide a performance benchmark, 
the real-walking subjects performed this same task while walking in the real OSC. 

Upon completing the training phase the VR subjects and the map subjects again completed the simulator sickness test. 

In the transfer phase, the subjects attempted to find the shortest path to the eight destinations in the real OSC. The 
experimenter named a destination and the subject sought only that destination until it was found. The subjects were asked 
to find the destinations in an order that differed from the order in which they saw them in the training phase. This was 
done to force the subjects to find new routes and truly navigate rather than just recall the path followed during training. 
The time and distance taken to find the destinations were recorded. 

Results 

In the orientation task, the mean absolute angular error from the attempts to point back to the origin of the tour (± 1.96 
standard error of the mean) were 33° (±14°), 71° (±14°), and 66° (±8°) degrees for the walking, VR-walking, and 
joystick groups, respectively. An ANOVA showed that there was a difference amongst the groups, F( 2,39) = 13.11, p < 
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.01. Planned comparisons indicated that the two VR groups performed worse than the real-walking group F( 1,39) 
26.21, p <.01, but they did not differ from each other, F(l,39) < 1. 

The mean distances required by each group to find the destinations are presented in Table 1. There are significant 
differences amongst the means, F(4,75) = 5.71, p < .01. In particular, the VR-walking group followed a shorter path than 
the joystick group, F(l, 75) = 3.99, p < .05. The VR-walking group was also superior to the no-experience group, 
F(l,75) = 7.16, p < .01, but the joystick group was not, F( 1,75) < 1. The route taken by the joystick group was farther 
than that taken by the real-walking group, F(l,75) = 12.94, p < .01, but the difference between the VR-walking group 
and the real-walking group was not significant, F(l, 75) = 2.83, .10 >p > .05. Similarly, the map condition did not differ 
significantly from the joystick group, F(l, 75) = 1.42, p > .10, or the VR-walking group, F(l, 75) <1. 



Real Walking 

VR-walking 

Joystick 

Map 

No Experience 

Mean Distance (m) 

643 

753 

886 

805 

928 

Mean Time (s) 

599 

805 

938 

909 

969 


Table 1. Performance in the Transfer Phase 

The mean times to find the destinations are reported for each group in Table 1. Following a square root transformation of 
the time data to alleviate heterogeneity of variance, an ANOVA revealed differences amongst groups, F(4,75) = 5.91, p < 
.01. Planned comparisons indicated that the VR-walking group required less time than the no-experience group, F( 1,75) 
= 3.15, p < .05, but the joystick group did not, F( 1,75) < 1. Both the VR-walking and joystick groups took more time than 
the real-walking group, F(l,75) = 6.77, p < .05 and F(l,75) = 14.71, p < .01, respectively. The differences between the 
map group and each VR group were not significant, both F(l, 75) <1. Finally, the VR-walking group did not differ from 
the joystick group, F(l, 75) = 1.78, p > .10. 


Ataxia Data 



—*— Map Lateral 
- - * - • Map Long. 

—•— Joystick Lateral 
Joystick Long. 
—±— VR-walk Lat. 

• ■ A- - ■ VR-walk Long. 


W Before After 


Figure 1. 

The data from the balance test, presented in Figure 1, revealed no effect of the VR training. The changes in lateral and 
longitudinal instability experienced by the VR groups were no greater than those experienced by the map group, all p > 
. 1 . " 
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38 

36 

34 

32 

30 


Sickness Questionnaire Data 



VR-walk Joystick 


□ Before 
■ After 


Map 


Figure 2. 

The sickness questionnaire data in Figure 2 show that the increase in symptoms following training was significantly 
greater for the VR groups than the map group. Using conversion equations provided by Knerr (1997), SSQ scores were 
derived from the questionnaire data and are presented in Figure 3 

Derived SSQ Scores 



□ Before ■ After 


Figure 3. 


along with scores from another VR system, (Witmer, Bailey, & Knerr, 1996) and the worst of 10 surveyed flight 
simulators (Kennedy, Lane, Berbaum, & Lilienthal, 1993). 


CONCLUSIONS 


This experiment documented that people can become quite disoriented in VEs and that the best way to learn about a 
space is to visit that space. However, VR proved itself to be an effective alternative to training in the real location. Spatial 
knowledge acquired in a VE can assist performance in the real world, but success is not assured. Navigation in VEs 
remains a human factors problem. 
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A walking interface to VR is helpful. It does not appear to help subjects remain oriented in the VE, but people transfer 
more knowledge to the real world if they use the walking interface rather than a joystick (at least in terms of finding 
the shortest path to a destination). 

The VR system used here caused simulator sickness, but the levels were not severe. They were comparable to other VR 
systems and some flight simulators. In terms of SSQ scores, this system seems to appear particularly nauseagenic, but 
there is reason to argue against this conclusion. This experiment was conducted during the winter, when many subjects 
were carrying thick coats^This may have caused higher responses to the sweating item in the nausea scale. Indeed, of the 
items used to derive the nausea score, the sweating item had the highest mean. 

Walking interfaces to VR appear to have significant military value. Users demonstrate better navigation performance if 
they use a walking interface rather than a joystick. This is important, not just for its own sake, but also because it 
indicates that users may be able to obtain non-visual information for navigation using the walking interface. Thus, they 
might be able to navigate when visual information is scarce due to smoke, night, lack of data for construction of a visual 
database, or when landmarks are destroyed. Furthermore, not being dependent on landmarks for navigation allows for 
improvising new routes out of sight of known landmarks. 

The walking interfaces available for obtaining these benefits can be compared on four dimensions pending further results 
on the role of proprioception in navigation. Ideally the interface will be inexpensive, simple, unobtrusive, and allow the 
user to physically turn in any direction. A conventional treadmill has several advantages. It is inexpensive, simple, and 
permits a normal gait, free from wires. However, it only supports movement in one direction. The omni-directional 
treadmill allows walking in any direction with a normal gait, free from wires, but it is expensive and complex. The 
rollerskate system developed by Iwata and Matsuda (1992) is simple, affordable, and allows movement in any direction, 
but it encumbers the user with wires and a harness. Slater, et al. (1995) developed a simple and inexpensive omni¬ 
directional interface, but it uses wired sensors that simply moves the subject at a pre-set speed. The feedback from 
walking naturally is not available. For this experiment a walking interface was developed that was low-cost, simple, and 
omnidirectional, but it does not allow a completely natural gait, and the user must wear wired sensors on each foot. 

Ultimately, a better understanding of human navigation is required to field an optimal walking interface. The kinesthetic 
and the vestibular systems are both involved, but their relative contributions are not known. Also unknown is the type of 
information derived from each system. Both can sense angular and linear movement. Once the nature of the information 
is known, and how important it is, new hardware can be targeted to provide only the information necessary, freeing the 
user from the costs and constraints entailed in providing the unnecessary information. 
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PERFORMANCE BASED DESIGN OF A NEW VIRTUAL 
LOCOMOTION CONTROL 


James Templeman 
Code 5513 - Naval Research Lab 
Washington, DC 20375, USA 

Tel: +1 (202)404-8847 - Email: templeman@itd.nrl.navy.mil 


Abstract 

The ability to simulate walking around in the environment is a key element missing from most of today's joint forces 
simulations. A number of sensor-based techniques are widely used to maneuver through Virtual Environments but they 
introduce artifacts into the interaction. Mechanical motion platforms have also been applied to surmount these 
difficulties, but they tend to exhibit different but equally troublesome side effects of their own. This paper examines the 
interrelationships between virtual motion control and other critical actions soldiers need to perform in VE. The goal is to 
allow the user to maneuver through VE in as similar a manner as possible to walking through the real world. If the 
interactions between different controls and sensory feedback can be made comparable to the interaction between actions 
in the real world, then there is hope for constructing an effective new technique. Human performance requirements are 
viewed from an analytical standpoint: pointing out the interactions between a full set of virtual controls that would allow 
the user to act, sense, and react to their environment. Candidate solutions are discussed as the analysis is developed. 
This has lead us to a promising new design for sensor-based virtual locomotion called Gaiter, introduced in this papei. 

THE TECHNOLOGY AND ITS USE 

Virtual locomotion is a control technique for allowing a person to move naturally over long distances in the Virtual 
Environment (VE), while remaining within a relatively small physical space. Ideally such an interaction technique will 
allow users to perform the same coordinated actions and move with the same agility in VE as they do in the real woild. 
Virtual locomotion is essential for constructing Individual Combatant Simulators where the soldier interacts diiectly with 
the surrounding environment. Previous military simulators have focused on vehicular weapons platforms where motion 
was controlled through the same kinds of control devices that are used to pilot the actual vehicle. Either actual or 
reduced cost versions of the controls used to pilot aircraft or drive a tank can be used in flight and tank simulators. This 
made it relatively straightforward to attain a high level of realistic interaction. For a locomotion simulator, the user's 
physical body is the control device and the user's avatar (representation of the user's body in VE) is the "vehicle" being 
controlled. This changes things considerably, calling for new kinds of control actions. 

DoD needs individual combatant simulators for both training and analysis. Such simulators can be used to train soldiers 
in team operations and tactical doctrine. They can be used to familiarize a group with a specific environment, and to plan 
and rehearse operations in that environment. An accurate VE model of an embassy could be used to train a team to carry 
out an embassy evacuation. For analysis, it is important to adequately simulate the performance of individual combatants 
in combined force simulations to make the simulation more complete and enhance its validity. The individual combatant 
is the "missing element" in today's military simulators. When this element becomes available, combined force simula¬ 
tions can be used to analyze and evaluate new tactics and operational plans. It will also allow new technology (for foot- 
soldiers, or jeopardized by foot-soldiers) to be studied with a comprehensive simulation. 

HUMAN FACTORS ISSUES INVOLVED 

A Framework For Analysis 

The goal is to find a virtual locomotion control technique as similar to actual locomotion as possible. But exactly how 
should it be similar? We approach this question through a careful analysis of the interrelationship between natural and 
control actions. 

Specifying A Control Technique 

It is useful to divide a control technique into two parts: the control action made by the user and the controlled effect 
produced by the computer system. Since in this case the control technique is used to mimic an action performed in the 
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real world, there is a corresponding natural action and natural effect . For virtual locomotion the natural action is 
perambulation (walking, running, sidestepping, etc.), the natural effect is real motion (physical displacement), and the 
controlled effect is virtual motion. There are many alternative control actions that can be used for virtual locomotion. 
They will be considered in terms of the analytical framework developed in this section. 

Interaction With Other Natural Actions 

Natural locomotion interacts with a wide variety of other actions. These actions include looking (directing both the head 
and eyes), manipulation (typically involving the hands, as in pointing a rifle), posturing (moving parts of the body, for 
purposes other than looking, manipulation, or locomotion), and thinking. Thinking is included because it is something 
the user does which interacts with other actions, including locomotion. Often the expression "cognitive load" is used to 
describe how thinking can limit or be limited by the performance of other actions. 

Components Of Locomotion 

Locomotion control will be divided into two components: control over the direction of motion (steering) and the rate of 
motion (speed). Steering may include a facility for sidestepping (i.e., moving in a direction other than where the torso is 
oriented). 

Simulating A Natural Capability 

The goal of simulated locomotion is to provide a control to maneuver through VEs as naturally as possible. We now 
have the terms to state this requirement a little more formally. (1) The control action should be similar to the natural 
action with respect to their intrinsic properties (constraints on movements, caloric energy expenditure, etc.). (2) The 
control action should interact with other actions (looking, manipulation, posturing, and thinking) in a similar manner to 
the way the natural locomotion interacts with other actions (e.g., freeing the hands for manipulation and allowing a 
normal level of coordinated head, arm, and leg movements). (3) The components of the control action (steering and 
speed control) should interact with each other as the components of the natural locomotion interact. (4) The controlled 
effect should be similar to the natural effect (i.e., the rate and precision of movement should match). This is clearly a tall 
order for any virtual locomotion control to meet, but it provides a basic set of criteria which can and should be applied to 
all candidate techniques. 

Compatibility With Turning The Body 

A great deal of compatibility with other natural actions can be achieved by having the user turn in VE by physically 
turning the body. This allows actions like turning the head, pointing the hand, or wielding a weapon to be performed as a 
natural coordinated motion. It also allows the reflexive orientation a person might have towards or away from a stimulus 
to be used effectively (e.g., turning one’s head to view the source of an unexpected sound). Another advantage of using 
the natural action of turning is to avoid a potential sensory conflict. People sense turning through their visual, aural, 
haptic, and vestibular systems. They build up a sense of their orientation in an environment through the temporal 
integration of this information. Peoples’ vestibular sense of motion contributes to their ability to navigate through virtual 
environments. 

We therefore prefer to treat turning the body as an action separate from locomotion, limiting locomotion to the control of 
the translational motion of the body through space. Steering is still required to control the direction of this motion. 
Although this eliminates the need for the control action to encompass turning, it imposes the constraint of having the 
control action be compatible with physical turning. The user should be able to perform virtual locomotion while turning, 
and the path traversed should vary accordingly. 

The body can be turned in several different ways. Most of the time, people step to turn: by lifting and planting their feet 
in a different direction on each successive step, the body turns to face a new direction. People also pivot on their feet to 
turn. Here the foot is turned while it continues to support a person’s weight. There are one- and two-footed pivots. 
Pivoting provides an means of turning the body that is more economical in terms of both time and space than stepping to 
turn, particularly critical in martial situations. 
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Compatibility With Real Motion 

It is highly desirable for the virtual locomotion control to be compatible with other sorts of postural movements like 
bending at the waist or crouching down. These may be thought of as special purpose actions - essential for certain tasks. 
A user might bend to look around a comer, or crouch to hide behind an embankment. It would be desirable if those 
motions could be made in a natural manner while performing virtual locomotion. We do not want users to learn an 
entirely new vocabulary of control actions just to allow them to move in different postures. 

It is also desirable to allow virtual locomotion to be intermixed with natural locomotion. The user could move forward in 
VE by making control actions or by taking actual steps (albeit over a limited range). Physical movements work well in 
VE, but must remain within tracker range and within the available space. 

Candidate Control Actions 

A variety of control actions can be performed to steer and set the speed of motion through the VE. This section will 
describe sensor based controls that have been employed for this purpose. The most obvious and widely used steering 
technique is that of pointing. With a vehicular control, the user’s body remains fixed in the vehicle and a rate dependent 
pointing device, like a steering wheel, is used to turn the vehicle. In an individual combatant simulator, the user can tuiii 
the body, so pointing can be direct. 

Steering Controls 

Head-based steering is probably the most widely used steering control for VE systems. It is economical because the 
same position tracker used to determine the user’s field-of-view is used to control the direction of motion. Othei 
advantages of head-based steering are that it "encourages" the user to look where s/he is going, and the hands are free for 
manipulation. The disadvantage is that looking and moving are no longer independent. The user cannot turn the head to 
look to the side while moving without altering the path. Bowman demonstrated this problem by asking users to walk to a 
position pointed at by an amow in VE [Bowman, Koller, & Hodges, 1997]. Users had to keep turning to look at the 
arrow as they moved toward the desired position; the act of looking interfered with their motion. The task was much 
easier to perform using hand-based steering. 

Hand-based steering techniques determine direction either from where the arm is pointed, where a hand-grip is pointed, 
or where a finger is pointed when an instrumented data-glove is worn. Hand-based steering frees the head for looking 
and allows the user to move sideways relative to the head or body. The disadvantage is that the control interferes with 
manipulation. The hand is occupied so that using the hand for other tasks leads to conflicts and interruptions. The user 
also must remain aware of the significance of where the hand is pointed. 

Torso-based steering frees the head for looking and the hands for manipulation, but it does not support sidestepping. 
Often people move in the direction where the front of their torso is pointing, but sometimes they do not. A soldier 
aiming a rifle across his chest may prefer to advance in the direction he is aiming. 

There are three approaches to lean-based steering: tilting the upper torso (bowing), shifting weight relative to one’s feet, 
and shifting one’s weight relative to a platform. All three kinds of leaning provide hands-free operation and can support 
sidestepping. Fairchild, Lee, Loo, Ng, and Serra, [1993] implemented a system based on shifting the user’s head position 
by tilting the upper torso (bowing). This control is incompatible with the user tilting the torso for other purposes. The 
second approach, shifting weight relative to the feet (i.e., to the basis of support) is of limited use because the continuity 
is broken when a user steps to turn. 

The third type of lean-based control is shifting one's weight relative to a platform. Max Wells at the University of 
Washington's HIT Lab has developed such a system [Wells, Peterson, and Aten, 1996]. Motion is controlled by moving 
one's body locally, relative to a central neutral position. When immersed wearing a head-mounted display, the user 
might lose track of where s/he stands with respect to the neutral position. The direction and rate of optical flow provides 
one indication of where the user is situated. A set of elastic straps attached to a ring around the user's waist gives haptic 
feedback, pushing the user back towards the neutral position. The directional coordinate frame is relative to a fixed 
external point in space, an unnatural condition that produces interesting side effects. This makes turning the body and 
controlling the direction of motion even more independent than they are with natural locomotion. The user may choose 
to move in one direction and then turn to face in another, making it as easy to "run" backwards as forward. This 
approach is incompatible with physical locomotion because velocity is controlled by relative position of the body. 
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Speed Controls 

Speed can be controlled using finger pressure, by the degree of leaning (when leaning is used for steering), by the rate at 
which the user steps tn-place, or by the degree of leg movement when the user steps in-place. Only the use of finger 
pressure will be discussed in this section. Often a pair of binary switches attached to a hand control are used to invoke 
either forward or backward virtual motion. This is a widely used technique, easy and inexpensive to implement. We 
have tried using pressure sensitive buttons and recommend their use because it provides a smooth way of varying speed. 
A major advantage of hand controls is that they work independently of head, torso, and leg movement and are thus 
compatible with a wide range of physical motions. The primary drawback of using the hand to control speed is that it 
interferes with using the fingers for manipulative tasks. As VEs become richer and engage the user in more complex 
tasks, they require a greater commitment of the user’s hands for manipulation. This is most evident for combat systems 
where the user needs to hold and aim a weapon both at rest and in motion. A second drawback of using hand controls 
with head-mounted displays is that the user cannot directly see how his hand holds the grip or how fingers touch the 
buttons. This limits the number of buttons the user can deal with. 

REVIEW OF THE STATE-OF-THE-ART 

This review will cover both mechanical motion platforms and a promising new sensor-based virtual locomotion system. 

Mechanical Locomotion Systems 
Uni-directional Systems 

A number of mechanical solutions have been proposed. One class of device is the uni-directional systems that limit 
movement to one direction and require a special control action for turning the virtual world around the user. An early 
system is the treadmill used at University of North Carolina in their Architectural Walkthrough System [Brooks, Airey, 
Alspaugh, Bell, Brown, Nimscheck, Rheingans, Rohlf, Smith, Turner, Varshney, Wang, Wber, & Yuan, 1992]. It uses a 
handle in front that the user twists to steer. They report that the treadmill is disorienting, probably because there is no 
true sense of turning which is helpful for route learning. Spinning the virtual world about the user produces an odd 
transient effect. Other uni-direction systems possibly suffer from the same disadvantage. Sarcos Research Corporation 
developed the TREADPORT, a large treadmill that allows the user to stand, kneel, sit, and lay prone [Youngblut, 1996]. 
The user is tethered for safety. OSIRIS, developed by the US Army's Night Vision Laboratory, is a stair-stepper device 
that can only be used upright [Lorenzo et cil, 1995]. It can present forward motion only; a hand control is used for 
steering. The Individual Soldier Mobility Simulator (ISMS) was developed by Sarcos Research Corporation in conjunc¬ 
tion with the Army Research Institute. The user stands on two robotic arm boot plates that create the sense of walking or 
running over different terrain, and ascending or descending stairs [Lorenzo et al ., 1995]. 

Multi-directional Systems 

Other mechanical systems allow the user to move in multiple directions. Sarcos Research Corporation's Uniport is based 
on a unicycle [Lorenzo, Poole, Deaso, Lu, Kekesi, Cha, Slayton, Williams, Moulton, Kaste, MacKrell, Paddison, Rieks, 
Roth, & Wodoslawsky, 1995]. The user remains in a seated position and turns around a pivot point by applying torque to 
the seat. This engages a motor to turn the seat at a controlled rate. (This system is at the borderline between a vehicular 
and locomotion control.) The user's hands are free to hold a gun or other objects. 

The Virtual Perambulator [Iwata and Fujii, 1996] uses a sliding motion of the feet to indicate walking. The user wears 
sandals with low friction film on the middle of the sole and a rubber brake pad at the toe. The user glides on a low 
friction surface by pushing the waist against a hoop that surrounds the user and sliding the feet. A position sensor 
attached to each ankle and contact sensors on the bottom of each foot allows the system to recognize the length and 
direction of each step to specify the movement in VE. An earlier version used roller skates instead of the low friction 
film. The placement of the hoop at waist level does not allow a user to hold an object by the side. 

The Omni-Directional Treadmill (ODT) developed by David Carmein of Virtual Space Devices Inc. [Darken, Cockayne, 
& Carmein 1997] allows the user to walk in any direction. The mechanism consists of a pair of conveyor belts nested 
one inside the other. Each track moves horizontally and the tracks are oriented perpendicularly to each other. The outer 
track has rollers on it so that it can transmit the motion produced by the inner track. The rollers can thus convey motion 
in any direction to the user's feet resting upon it. The active surface of the motion platform measures 50"x 50". A 
control system is used that seeks to keep the user centered in the middle of the platform. The system is very effective at 
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allowing a user to walk in a straight line in any direction. People have little difficulty accelerating along a straight path. 
Turning while accelerating can lead to a misalignment between the user’s direction of translation and the centering 
motion of the mechanism. This is apt to make the user lose his/her balance and stumble. Even turning m-place can be 
difficult because the tracking system registers the motion and compensates for it by moving the surface under the user's 
feet. Sometimes the dynamics of natural locomotion go beyond what the ODT can support. People can decelerate very 
quickly, coming to a full stop in a single step, and if they pivot while walking fast they can redirect their motion even 
more rapidly. Further, people perceive linear and angular acceleration using their vestibular system. A user immersed in 
VE, standing on a "perfect" motion platform would have the sensation that the vestibular perception of linear acceleration 
were "turned off'. This alone will make it difficult to perform certain actions, even if the dynamic response of a motion 
platform were perfect. 

First Attempt Using Walking-In-Place 

The first virtual locomotion system we implemented and tested used a combination of torso-based steering and step- 
based speed control. Sidestepping was not supported. A position tracker was attached to the users waist and pressure 
sensors were mounted under the ball and heel of each foot on the insoles of their shoes. The basic idea was to initiate 
virtual motion when the user took an in-place step and use the rate of stepping to control speed. The system also had to 
be able to distinguish between stepping to turn and stepping in-place to initiate virtual motion. We made the mistake of 
putting the burden of making this distinction on the user. Forward and backward virtual motion were indicated by 
pushing off the ball or heel of the foot, respectively. Flat footed steps produced no virtual displacement and were to be 
used for stepping to turn. The user could also turn while advancing in VE by pushing off the ball of their foot while 
stepping to turn in-place. This was easy to do when the user concentrated on it, and the computer had no problem rec¬ 
ognizing these distinctions. But users tend to get a little "sloppy" when they started thinking about something other than 
how they stepped. Although ninety percent of their steps indicated their intention, the remainder is enough to spoil the 
technique. Unintentionally movement, or missing a step is quite disconcerting. 

Gaiter: Our New Locomotion Control 

We developed a second generation locomotion control to allow the user to more naturally control their movements 
(without having to concentrate on their footsteps) and to support sidestepping. We call the new system Gaiter. Its design 
is based on generalized classification of human gait that encompasses sidestepping and in-place stepping as well as 
forward stepping. 

The user walks in-place to walk through VE. The user can move in any direction by rocking the knees. Forward virtual 
motion is achieved by simply walking in place. In this case the knees rock forward then back. To take a virtual sidestep 
to the right the user lifts his right knee up to the right and then drops it back down (to the left); the left foot should be 
raised and lowered on alternate steps, but need not rock horizontally. Backward virtual steps are taken by rocking the 
knees backward first then forward. The user can rock the knees along a diagonal path to move diagonally through VE. 
The direction and degree of virtual motion is determined by direction and degree with which the knee moves. Virtual 
displacement is a function of both how far the knees swing and the stepping rate, as physical displacement is a function 
of both stride length and stepping rate. 

Gaiter employs a hybrid sensor system. Position trackers placed on the knees register knee motion. Pressure sensors 
placed on shoe insoles under the ball and heel of each foot register ground reaction forces. The pressure sensors help 
determine the timing of each step, punctuating knee motion. Pattern recognition software distinguishes between steps to 
turn, in-place steps, and actual steps. Since the system can distinguish between them, it can allow the user to intermix 
virtual and actual steps. Virtual steps move the user in the VE while remaining stationary in the real world. 
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Gaiter: a new virtual locomotion control being developed at NRL. The user walks in-place to walk through VE. 
Steps are characterized using a combination of position trackers and pressure sensors 

Actual steps move the user the same distance in both the virtual and real world. Actual steps provide the most natural 
and accurately judged movements but their range is limited, so virtual steps are used to cover large distances. 

Features Of Gaiter 

Many of the "realistic” features of Gaiter derive from its adoption of leg motion as the control action. It operates in the 
appropriate coordinate frame of reference: the direction of knee movement with respect to the surrounding environment. 
Virtual motion may be made in any direction. Sidestepping is an inherent part of the technique. It is compatible with 
turning the head and body, using the hands for other tasks, a variety of postural movements, and actual walking. The 
system can be tuned to preserve metrics between physical and virtual space. Knee movement can be scaled and mapped 
into virtual displacement. The system is responsive; the pattern recognition system does not have to wait for the gesture 
to be complete before responding to knee movements. Virtual motion occurs as the knees rock. The equipment used to 
implement Gaiter is compact and relatively inexpensive. A wireless version can be built today, albeit at greater expense 
than using tethered sensors. 

The basic limitation of Gaiter is that walking and running in-place are not the same as actual walking and running. They 
use somewhat different muscle sets in somewhat different ways. The user does not physically accelerate along the path 
of motion or build up momentum by running faster in place. (It may be possible to simulate the latter effect by 
increasing virtual velocity as successive strides are made, but this may make the requisite deceleration seem more artifi¬ 
cial.) It will be possible to tune the system to match particular attributes of natural locomotion (e.g., perceived velocity, 
natural cadence, caloric expenditure, etc.) but it is unlikely that one set of tuning parameters will satisfy all criteria. 

Experimental Evaluation 

We are currently running pilot studies comparing three different sensor-based virtual locomotion controls: head-based, 
hand-based, and leg-based (Gaiter). We have started by using the set of locomotion tasks described as part of VEPAB 
(ARI's VE Performance Assessment Battery [Lampton, Knerr, Goldberg, Bliss, Moshell, & Blau, 1994]). We also have 
models of the Quantico Village database developed by Paradigm, Inc. for the Team Tactical Engagement System for 
NAWC/TSD and the Ex-USS Shadwell ship interior developed Dave Tate at NRL [Tate, Sibert, & King, 1997]. This 
gives us a variety of interesting virtual spaces to walk through. We are tuning the current version of Gaiter to make it 
match peoples’ natural strides. The tasks we will use in the comparative experiment range from simple locomotion down 
hallways and through rooms, to the more complex task of clearing a building by pointing and clicking at targets while 
minimizing one's exposure to them. Our hypothesis is that people will perform tasks differently using the three different 
controls and the sequence of actions (including looking, moving, and manipulation) they perform using the leg-based 
control will best match the sequence of actions used to perform the task in the real world. 
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LIMITATIONS OF THE TECHNOLOGY 

There are many limitations to the sensors, processing systems, and displays used to construct today's VE systems. 
Spatial position trackers have a limited range, are sensitive to metal and interference from other sources, and are often 
tethered. Currently available head mounted displays light weight enough to wear while walking offer a limited t.eld-ot- 
view (typically 50-degrees), a limited display resolution (typically 640 x 480 pixels), a fixed depth of focus, and are often 
tethered. Current rendering systems provide limited realism of real-time imagery. The lag due to sensor transmission, 
image generation, and display contributes to motion sickness [DiZio & Lackner, 1992], 

There are clearly limitations to the physical interaction provided by VE systems. In terms of locomotion the ground 
surface is almost always flat, with fixed surface characteristics. This could be improved by employing a hydraulically 
tilted floor with adjustable surface compliance. The remaining omission would then be the absence of the collision 
forces made contacting walls and other raised surfaces. The challenge will be to add these features while keeping the 
system safe enough to use without falling. 


Consequences Of Unrealistic Interaction 

What happens when the requirements for interactive realism stated earlier do not hold; when similarity of (1) action, (2) 
interaction (3) component actions, or (4) effect are not fully attained? The actions must be the same for the user to 
develop the right set of coordinated reflexes. The capability (effect) must be the same for users to learn when to apply it, 
to incorporate it into strategies, and to avoid changing their strategy to compensate for its absence. Compensation is bad 
because it means that users are adopting different practices and approaches in VE than they would use in the real world. 

In terms of isolated capabilities, capabilities can be lost in VE, new capabilities can be gained, and the control action can 
alter the way a capability is used. All of these cases are departures from realistic task performance. If a control lets the 
user effortlessly move at a rapid speed, the user might carry out rehearsal missions in VE that would be impossible to 
accomplish in the real world. In terms of interacting capabilities - the control action can preclude other capabilities (e.g., 
not being able to stop quickly may discourage rapid motion), the control action can free the user to perform extra actions 
(e g a technique that supports backwards motion while the user stands at rest would allow the user to aim a rifle 
accurately while retreating from the enemy), and the control action can alter the way other capabilities are used (e.g., not 
being able to advance in a crouching posture limits the kind of ground cover one can hide behind). 

PROJECTS USING THIS TECHNOLOGY 

Many applications would benefit from the ability to walk through VE in a realistic manner. In the Team Tactical 
Engagement Simulator (TTES) developed by the Naval Air Warfare Center Training Systems Division (NAWC/TSD) 
and Southwest Research, military personnel practice urban warfare tactics [Horey Fowlkes, & Reir, 1995; Goodman, 
Porter, & Standridge, 1997]. The Naval Research Laboratory's Information Technology Division developed a VE of 
portions of the Ex-USS Shadwell, the Navy's full-scale fire research and test ship, to test the feasibility of using 
immersive VE as a tool for shipboard firefighting training and mission rehearsal [Tate, Sibert, William, King, & Hewitt, 
1995]. Navy firefighters who practiced in the three dimensional model that included fire and smoke effects to simulate 
actual firefighting conditions performed better than those who only practiced using the ship's diagrams [Tate et al„ 1997], 
The University of North Carolina's Architectural Walkthrough System allows designers to explore a building design 
before its construction [Brooks et al ., 1992]. 

DIRECTIONS FOR THE FUTURE 

We will continue to refine Gaiter. There are different ways of mapping the characteristics of in-place stepping to virtual 
steppine. We have initially adopted a linear mapping between the displacement of the knee while stepping in-place to 
the displacement of a virtual step. Further studies will be carried out to explore how well this matches natural stride 
lengths and stepping tempo. A variety of criteria will be used to measure the fit including the user’s perceived rate of 
motion, the energy expended, or subjective effort made to move at different speeds. 
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We are developing an avatar driven by Gaiter. The avatar has two basic uses. It allows the user to see how his body fits 
into the VE, giving the user a better sense of immersion [Slater Usoh, & Steed, 1995]. An avatar also lets other people 
see the user’s pose and location in VE. The challenge of developing an avatar driven by Gaiter is to match the control 
actions (in-place stepping) to the controlled effects (virtual stepping) in as natural a way as possible. The match will not 
be perfect and it remains to be seen how tolerant users will be to different kinds of mismatch. 

CONCLUSIONS 

There are many interactions between perceptual-motor systems and candidate locomotion controls involving looking, 
manipulating, turning the body, posturing, and thinking. These interactions limit the most promising approaches to those 
using the legs to make actions similar to natural locomotion. Our first attempt at building a leg-based control put too 
high a burden on the user to make special kinds of steps. We combined position trackers on the legs with the insole 
pressure sensors to create a walking simulator called Gaiter. Gaiter provides a consistent means of taking virtual steps in 
any direction. We will test our new virtual locomotion technique against two common sensor-based techniques: head- 
based steering and hand-based steering, both using finger pressure to control speed. The controls will be evaluated in 
terms of how similar peoples' actions in VE are using the control compared to peoples' actions in the real world. 
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ABSTRACT 

Head mounted display systems typically present imagery to the user focused either at infinity (collimated light into the 
eyes), or alternatively at some nominal finite distance in the order of 11 - 18 feet (diverging light into the eyes). When 
the imagery presented by an HMD is focused at a finite distance, the right and left eyes are sometimes intentionally set to 
converge to that same distance. Alternatively, the eyes are often left viewing parallel to one another. In some HMDs the 
user is permitted to select his or her own preferred focus distance. There appears to be an advantage in improving depth 
perception in an HMD simulation environment by altering in real time the focus or apparent object distance to match the 
distance of the principal object or objects being viewed at that time by the user. An eye tracker may be employed to 
determine where in the scene the user is looking, and the data is fed back to the computer to perform, as appropriate, the 
refocusing task based on the known distance of the object being looked at. Further realism improvement is likely by 
changing °the convergence as well. This paper will discuss the relative merits of altering the accommodation and 
convergence as well as the means for accomplishing the refocusing task in the HMD viewing optics rapidly, in real time, 
and without otherwise altering the image quality or magnification. The net goal is to improve the overall realism of the 
simulation to the user. 

INTRODUCTION 

Head-mounted displays (HMDs) have found widespread use in a variety of applications. Their application for pilot 
training has been especially successful because they are much less expensive and less dangerous than actual flying, and 
far less expensive to implement than large dome simulators. 

To achieve the best results in training, the training environment presented by the HMD should be as close as possible to 
the actual environment experienced by the pilot. One of the most important improvements in HMDs is the accurate 
simulation of depth and three-dimensional images. 

The need for high quality three-dimensional imaging in flight simulation and other applications has driven the 
development of various technologies (Okoshi, 1980)). Many of these technologies are fairly complex (i.e. Williams, 
1988) and are still far from implementation. As a result the only technique currently being used in such applications is 
dichoptic (separate image to each eye) presentation of binocular disparity. This paper describes preliminary efforts at 
OPTICS 1 to improve the realism of HMDs and to determine the merits and viability of adjusting in real time the 
accommodation or apparent object distance as well as the convergence in an HMD. 

SPECIFICATIONS AND REQUIREMENTS 

The HMD specifications to which we will work are summarized in Table 1 below. These data represent our best effort at 
describing a viable and achievable HMD implementation with good performance. 


Paper presented at the RTO HFM Workshop on “The Capability of Virtual Reality to Meet Military 
Requirements”, held in Orlando, USA, 5-9 December 1997, and published in RTO MP-54. 
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Parameter 

Specification 

Full diagonal field of view 

40 degrees 

Exit pupil diameter 

7mm minimum, 10mm goal 

Eye relief 

25.4mm minimum 

Exit pupil location 

Always at eye pupils 

Interpupillary distance, nominal 

62 mm 

Range for interpupillary distance 

52 - 72 mm 

Distortion 

3% maximum 

Image blur within horizontal inscribed circle 

2.5 minutes of arc maximum 

Image blur to comer 

3.5 minutes of arc 

Spectral band 

Visual 

Spectral weighting 

Photopic eye response 

Minimum accommodation distance 

15 inches 

Maximum accommodation distance 

Infinity 

Convergence at infinity 

Zero 

Convergence at 15 inches 

9.3 degrees 

Speed of change in accommodation 

<0.6 seconds 

Magnification change over accommodation range 

4% maximum 

Eye tracker 

Yes 


Table 1. 


VISUAL EFFECTS AND MERITS OF VARYING ACCOMMODATION AND CONVERGENCE TO 
ENHANCE REALISM 

Depth Perception in a Natural Environment 

The human mind uses a wide variety of clues to determine distance and depth. These clues include angular extent of an 
object, occlusion by other objects, perspective, convergence, and accommodation. Each of these clues need to be 
carefully considered in the complete implementation of an HMD system. However, only convergence and 
accommodation can be addressed in the physical and optical design of an HMD system, so these will be the only topics 
addressed in this paper. 

Convergence is the rotation of the eyes towards an object. As a person looks at an object, the eyes rotate so that the 
object is imaged onto the fovea of the retina which is the small region of highest visual acuity. This rotation means that 
the centerlines of the eyes intersect at the object. Convergence is measured by the total included angle between the eye’s 
centerlines. This angle between the eyes is one of the most important visual clues to depth. Unfortunately for HMD 
designers, this angle is a function of the user’s interpupillary distance or IPD. 

Accommodation is the refocusing of the eyes to the distance of the object being viewed. 

In a natural viewing environment, accommodation and convergence work together to aid in determining distance. The 
distance at which the eyes are focussed is the same as the distance at which the eyes’ centerlines intersect. 

Depth Perception in an Artificial Environment 

HMD designers must try to approximate the natural environment depth perception as well as possible. However, they are 
subject to several constraints which should also guide their design considerations. First, because of the inter-relationship 
between the IPD and the convergence angle, an exact match between accommodation distance and convergence angle is 
possible for only a small set of cases. Second, because objects of varying distances may be represented on the same 
viewing screen, some mismatch between accommodation distance and represented distance is inevitable. 

Two measures describe the degree to which an HMD differs from the natural viewing environment. The first measure is 
called “accommodation error” and is a measure of the difference between the distance at which the eye is focussed and 
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the distance at which the object is represented. The second measure is called “convergence error” and is a measure of the 
difference between the convergence angle required by the displayed objects distance and the convergence angle 
presented by the HMD. 

Many difficulties occur when the convergence error and accommodation error are too large. These difficulties include 
loss of realism, headaches, double vision, blurred vision, and, in extreme instances, even nausea. For small errors, these 
difficulties tend to ease as the duration of HMD use increases. For larger errors and longer duration of HMD use, 
however, these difficulties can persist even after the used of the HMD. 

Determining the acceptable errors in accommodation and convergence angle is an active area of research. The HMD 
design community should probably never expect to obtain definitive, universally applicable results for the acceptable 
errors for several reasons. First, this research is, by definition, very subjective and dependent on the particular 
experimental parameters. Second, different situations will have different acceptable errors; the requirements on a HMD 
for a military flight simulator are very likely to be different from those of an inexpensive video game. Third, everyone’s 
visual system is different; a wide variety of common eye conditions can cause people to be unusually sensitive or 
unusually lax in their tolerance to convergence error or accommodation error. Finally, people have a wide variety of 
tolerances to non-ideal situations, due merely to their personalities, moods, and temperaments. 

To precisely investigate visual effects and merits of varying accommodation and convergence, it is important to precisely 
define our terms. Accommodation error is 

A = l/s 0 -1 /sf, 

where is the object distance to be represented, Sf is the distance at which the HMD is focussed, and A is the 
accommodation error. Accomodation error is usually expressed in diopters. (1 diopter = 1/meter) The ideal 
convergence angle is 

CA idea l = IPD/Sg, 

where IPD is the interpupillary distance, and CA idea { is the ideal convergence angle. The convergence angle error is 

CA ~ CA real ~ CAi dea i, 

where C a is the convergence angle error, and CA rea i is the convergence angle presented to the user. Convergence error 
can be expressed in either degrees or prism diopters. (1 prism diopter = 0.01 radian = 0.57 degrees). 

Nevertheless, it is helpful to investigate the requirements of an HMD for various assumptions of convergence angle error 
and accommodation error. 

Figure 1 shows how the convergence and accommodation interrelate, and how errors in both can happen. 
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Figure 1 - Simulating Distance in a Real HMD ^TpH^ed 

• Exact accommodation and convergence can only be obtained 
for a small set of circumstances 

• Accommodation error: eyepieces focussed at wrong distance 

• Convergence error: boresights intersect at wrong distance 



accommodation error 
error 



The top line in the figure shows that, for an acceptable accommodation error of 1/16 diopter, 16 focus positions would be 
required to represent the range of object positions; for object positions between 0.5m and 2m, a near-continuum of 
positions is required. The bottom line on the figure shows that, for an acceptable accommodation error of 1/3 diopter, 
only three focus positions are required. Figure 3 shows the required convergence angles for object distances from 0.5 
meter and above. The two curves are for convergence errors of 1.5 degrees and 0.2 degrees. 


Figure 2 shows the required focus positions for object distances from 0.5m to infinity for two acceptable 
accommodation errors. 


Figure 2 - Accommodation and HMD Design 



• Focus positions to represent object distances from 0.5m to infinity: 

• for 1/2 diopter accommodation error, only two focus positions 
are required 

* for 1/16 diopter accommodation error, 16 positions are required 
(continuous focus might be better) 
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Figure 3 - Convergence in HMD Design 
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Convergence angles to represent object distances from 0.5m to Infinity 

• for 1.5 degree convergence error, six convergence angles are required 

• for 0.2 degree convergence error, 12 convergence angles are required 
(continuous adjustment might be better) 


Accommodation Methods 

A wide variety of accommodation methods are feasible. These methods include, in no particular order. 

1. Translating the lens assembly 

2. Translating the display device 

3. Translating a sub-group of lens elements 

4. The Alvarez asphere, bifocal/multifocal lenses 

5. Bifocal or multifocal lenses 

6. Liquid filled lens 

7. Liquid crystal lens 

8. Adaptive optics 

1 Translating the lens assembly relative to the display device changes the accommodation distance from infinity 
to a closer image distance. Presenting images at infinity and at 0.5m requires moving the lens assembly about 3mm 
for a 35mm focal length lens. This method of accommodation has several advantages, most importantly, it is easy to 
implement. This method has some disadvantages, too: it requires moving parts. A potential disadvantage is that the 
exit pupil location where the eye is located may move causing vignetting or clipping of light at off axis field 
positions. 

2. Translating display device switches from a distant image distance to a close image distance by moving the 
display device from the back focus of the lens to a position inside the back focus. This is similar to translating the 
lens assembly. It has the advantage of not changing the stop position if the viewing optics is designed to be 
“telecentric” at the display. Unfortunately, it has a serious disadvantage in that display mounts are usually fairly 
complex and would be difficult to move. 

3. Translating a sub-group of lens elements within the lens assembly switches from a distant image distance to a 
close image distance by changing the lens assembly’s focal length much as in a zoom lens. This method has the 
advantage that it can be easy to implement and that the moving mass can be very small. Its disadvantages include 
more complex lens housing and the inherent disadvantages of moving parts. 
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4. The Alvarez asphere switches from a distant image distance to a close image distance by moving special 
aspheric surfaces perpendicular to the optical axis. A similar method was used in the auto focus mechanism in the 
Polaroid Spectra camera. This method has several advantages: only a small mass must be moved, the center of mass 
does not change, and only a small motion is required. Further, this motion is linear. This method also has several 
disadvantages: the aspheric surfaces are difficult to fabricate because they are not rotationally symmetric, and the 
effect on image quality can be unacceptable. 

5. A bifocal or multifocal lens can be constructed such than some regions of the lens form a distant image and 
other regions form a close image. This method is particularly attractive for applications which only require two 
focus positions, and because no moving parts are required. Disadvantages of this method include lowered resolution 
and fabrication difficulties. 

6. A liquid filled lens has been used in some applications requiring a variety of focus positions. The liquid is held 
between compliant thin windows that form the outer optical surfaces. As liquid is pumped into or out of the lens, the 
compliant walls change shape, causing a change in focal length of the device. This method has the advantages of 
having no moving optical parts and of the potential for quick adjustment times. This method has the disadvantages 
of a limited accommodation and all the troubles that come with the use of a liquid. 

7. Liquid crystal lenses have been used in many applications requiring a quick adjustment in focal length. In a 
liquid crystal lens, the liquid crystal is placed inside of a lens-shaped cavity. When an electric field is applied to a 
liquid crystal, the molecules tend to align. Their uniaxial or biaxial symmetry creates a phase retarder. The liquid 
crystal lens is made without polarizers. Rather than create a quadratic phase variation, researchers have made liquid 
crystal Fresnel lenses. As voltage is applied to the liquid crystal, its index changes, causing a change in the focal 
length of the lens. This method has the advantages of having no moving parts, the potential for very fast adjustment 
speed, and maintaining the correct pupil position. This method has the disadvantage of having a limited 
accommodation, as well as having all the disadvantages of using a liquid crystal. 

8. Adaptive optics are also often used in changing the phase profile of light. Adaptive optics are lenses or mirrors 
which change shape using a variety of actuator methods. This method has the advantage that a continuous 
adaptation can be provided. However, adaptive optics usually suffer from the disadvantages of being difficult to 
control; furthermore, they are a new technology and therefore tend to be expensive. 

CONVERGENCE METHODS 

A wide variety of convergence methods are also available. These methods include, in no particular order, the following: 

1. Video centering 

2. Translating the display 

3. Tilting the entire assembly 

4. Optical wedges in front of the 
eyes 

1. Video centering achieves convergence by electronically moving the center of the scene away from the center of 
the display device. For example, when simulating the convergence of an object at infinity, the images will be moved 
towards the nose to create the desired convergence angle. This method has the advantage of having no moving parts 
and allowing quick convergence times. Its disadvantages include more complex electronics, rotation about the pupil, 
and the fact that the display must in effect be larger than what is required for any given convergence situation. 
Figure 4 shows a graphical representation of the video centering methodology. 
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Figure 4 


2. Convergence can also be achieved by translating the entire display devices towards the nose. This is often a 
problem as the electro-mechanical implementation may be difficult. 

3. The entire viewing optics assembly including the optics and the display can be tilted to match the desired 
convergence angle. Ideally this tilt angle should be at about the eye pupil location so as to avoid vignetting or other 
bad optical effects. While this is a difficult implementation task, it correlates directly with the desired result, making 
it somewhat attractive. 

4. We can locate optical wedges in front of the eye so as to vary the magnitude of convergence presented to each 
eye. If two optical wedges or weak prisms are counter-rotated we can achieve a variable degree of convergence. The 
disadvantages include the mechanization complexity as well as lateral color or color fringing around objects. A 
liquid filled variable wedge is also feasible. 

EXAMPLE OF A CANDIDATE IMPLEMENTATION 

In order to give a realistic example of a viable implementation, we have modified our PT-Ol HMD product in order to 
allow for a rapid change in both accommodation and convergence from infinity to a distance of approximately 15 inches. 
This HMD is a commercial product used in video inspection, cinematography, and other professional areas. Its field of 
view is 27.5 degrees diagonal and thus not ideal for a simulation application, however we used it here due to its 
availability. 

The convergence change was accomplished by rotating the viewing optics and display assemblies in front of each eye by 
approximately 4.6 degrees towards the nose, and the accommodation was accomplished by refocusing each of the 
viewing optics lens assemblies by 3.8 mm towards the LCD display device. As it was not possible to rotate the 
assemblies about the ideal eye pupil location, the performance is not ideal. However, it does demonstrate the feasibility 
and the overall performance of such an implementation. 
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Figure 5 shows the lens design for the HMD as well as its optical performance in the form of geometrical spot diagrams. 
We show that the imagery at the very corner of the field of view is in the order of 5 arc minutes rms blur diameter which 
is reasonable for this application. We also show the lens assembly and display device rotated towards the nose by 4.6 
degrees. 
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SUMMARY AND CONCLUSIONS 

The literature clearly shows the importance of providing the cues of accommodation and convergence in an HMD when 
used in a simulation environment. This is especially true when the user is in a simulated cockpit or similar environment 
where imagery of close objects must be simulated as well as imagery of objects at infinity and/or other distances. 

We have shown the accommodation and convergence demands for a typical HMD as used in a simulation environment. 
It is important that the accommodation and convergence demands both be appropriately coupled. Eight different 
methods for implementation of a variable accommodation and 4 methods for variable convergence in an HMD were 
presented. The different methods differ from one another in complexity, performance, and other factors. Additional 
work will be directed towards identifying the optimum system for implementation. 
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CAPABILITY OF VIRTUAL ENVIRONMENTS TO MEET MILITARY 

REQUIREMENTS 
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Essex Corporation, Orlando, Florida, USA 
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The DoD and NASA are considering virtual environment (VE) technology for use in forward deployable and remote 
training devices. Yet, many of these VE devices, particularly those which employ helmet-mounted displays, have an 
adverse effect on users, eliciting motion sickness and other sequelae (e.g., Pausch, Crea, & Conway, 1992; Kennedy, 
Lane, Lilienthal, Berbaum, & Hettinger, 1992). These symptoms, now called cybersickness (McCauley & Sharkey, 
1992), could retard development of VE technology and limit its use as a training tool. 

Motion sickness is known to be polysymptomatic and in scoring self-reports we have found there to be reliably different 
profiles of sickness in simulators, at sea, in space, and in VE (Kennedy, Lane, Berbaum, & Lilienthal, 1993). 
Furthermore, recent research in our laboratories implies that cybersickness may involve multiple functional pathways. 
The first pathway is related to ill-effects upon the autonomic nervous system or ANS (Money, Lackner, & Cheung, 
1996). According to sensory conflict theory (Reason & Brand, 1975), the ANS is provoked when sensory inputs from 
the visual, auditory, vestibular, or somatoceptors are uncorrelated or incompatible. This is the case when one is exposed 
to the certain sensory rearrangements in a virtual environment. Such rearrangements can trigger the "emetic brain 
response" (Oman, 1991), causing vomiting, perspiration, nausea, pallor, salivation, and drowsiness. 

Slower effects might include the sopite syndrome. This syndrome may involve a sleep-related pathway. Student Naval 
aviators, referred for airsickness, show that sopite syndrome can occur during military flight training in an individual 
who is "immune" to airsickness and debilitate an individual long after he has recovered from overt airsickness. 

When the ANS pathway is triggered it can also lead to performance decrements. Performance problems can arise due to 
lowered arousal and decreased concentration or because individuals experiencing ANS symptoms will attempt to 
minimize ill-effects by modifying their behavior (Hettinger, Kennedy, & McCauley, 1990). 

It has also been shown that individuals restrict their head movements while using helmet-mounted displays (Hennesy, 
Sharkey, Matsumoto, & Voorhees, 1992), and that inhibition of head movement "learned" in the trainer transfers to 
performance in the actual helicopter. It is clear that if VE systems are to be effective training devices, such negative 
transfer must be avoided by obtaining a better understanding of the ANS mechanism. 

Another cybersickness pathway involves adaptation within the central nervous system (CNS). In a well-respected theory 
of CNS functioning, von Holst (1954) argues that motor impulses (or efference) leave images in the CNS which are 
compared to the reafference generated by the effector (i.e., the stimuli resulting from one’s own muscular activity). 
Normally, the efference and reafference images match, thus leading to coordinated muscular activity. When something 
causes a mismatch, coordinated activities may be degraded (e.g., degraded hand-eye coordination) because of the 
limiting effects of the reafference. This mismatch can also provoke the ANS pathway, producing motion sickness. Held 
(1965) and Reason (1978) provide evidence to suggest that the effects of the ANS pathway discussed above can be 
overcome by facilitating CNS adaptation to these neural mismatches. Regardless of one's opinions about a neural 
mismatch hypothesis, it is clear that adaptation is fostered when "mismatches" are sufficiently regular, users have control 
over their movements, and/or they receive a sensory response (e.g., visual, vestibular, or proprioceptive) to their actions. 


* Currently at: RSK Assessment Inc., 1040 Woodcock Road, Orlando, FL 32803, USA 


Paper presented at the RTO HFM Workshop on “The Capability of Virtual Reality to Meet Military 
Requirements ”, held in Orlando, USA, 5-9 December 1997, and published in RTO MP-54. 
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Thus, if the sensory rearrangements of the VE are sufficiently small, gradual, regular, and "reafferent", the plasticity of 
the CNS more frequently allows adaption. This adaptation is characterized by: a decline in the initial response to an 
altered stimulus; development of an altered, often compensatory response following prolonged exposure to the change; 
and a continuation of the adapted response (i.e., an aftereffect) once the stimulus is removed (Dolezal, 1982; Parker & 
Parker, 1990; Welch, 1978). Such post-adaptation aftereffects have been known to persist for several hours after system 
exposure (Baltzley, Kennedy, Berbaum, Lilienthal, & Gower, 1989; Crosby & Kennedy, 1982; Ungs, 1987) and may 
occur without any symptoms of motion sickness discomfort except disorientation. 

Along the first ANS pathway of cybersickness, if an individual is sick he/she will perceive this discomfort and modify 
his/her behavior to minimize the ill-effects. Along the second pathway, however, if adaptive changes occur individuals 
may be unaware of the CNS modifications. Since the two pathways follow a different time course, it is possible for both 
effects to be present; or either may be present without the other. We will report some preliminary evidence for these 
predictions. 

Much of the research to date on self-reports of sickness symptomatology has used a measure called the Motion Sickness 
Questionnaire (MSQ). The MSQ was developed more than thirty years ago for studying the causal mechanisms 
underlying motion sickness. A paper-and-pencil version of the MSQ was later tailored to collect data from simulator 
participants, and scoring norms were developed using a calibration sample of 1600 simulator exposures. The Simulator 
Sickness Questionnaire (SSQ) was developed by Kennedy, Lane, Berbaum, and Lilienthal (1993) based on empirical 
studies. The symptom checklist portion of the SSQ is comprised of 28 symptoms, most of which are rated on a four-point 
ordinal scale with anchor points at "none," "mild," "moderate," and "severe". The items on the checklist have 
considerable overlap with the signs and symptom checklist of Lackner and Graybiel. Scoring our checklist by their 
scoring key versus ours produces Total Scores from the two methods which correlate (r > .83) and we showed a figure of 
our database of two dozen simulators that we use as norms and which are scored also with the Lackner scoring key. We 
also presented additional normative data for 8,000 exposures from simulators to be compared with data from >450 virtual 
reality exposures. 

In addition to the algorithm for Total Score, the original data base of 1000+ exposures was also used to carry out a factor 
analysis of simulator sickness symptoms (Kennedy et al., 1993). The symptoms from this large group of exposures 
revealed three clearly defined factors: nausea and neurovegetative complaints (N), oculomotor disturbances (O) and 
disorientation effects (D) (cf. also Lane & Kennedy, 1988). These three clusters fit nicely with theoretical descriptions of 
motion sickness (e.g.. Money, 1970; Reason & Brand, 1975) and, for those of us who have personally experienced 
motion sickness, these three factors have obvious face validity. Compared to the Lackner and Graybiel scoring method, 
the nausea subscale of our scoring system correlates higher (r = .88) with their total score. The total score correlates 
much lower (r = .62) with our disorientation and oculomotor scoring keys. From this factor analytic approach we have 
therefore hypothesized that if sufficient individuals are studied, it may be possible that the distribution or configuration of 
the three factors may turn out to be consistent within a given simulator and different between simulators. If so, then 
perhaps this might provide a method whereby the many different causes of simulator sickness can be delineated. 
Therefore, while Total Score differences in simulators may index the level of the problem, differences in profile or 
configuration, REGARDLESS OF LEVEL OF SICKNESS, may signal the nature of the cause of sickness in that 
simulator. 

In a series of 3 experiments, using 3 different scenarios displayed over two different helmet mounted systems, subjects 
(N = 75) were exposed to 20-40 minutes of VE. We found in all three studies: 

1) pre post differences in self-report of motion sickness were statistically significant; 

2) pre post motor effects (past pointing and/or posture changes) were significantly different; 

3) although changes in both types of variables were reliably observed, the sickness severity and size of side effects 
themselves were not correlated. That means that persons who were sick may or may not have had postural effects and the 
converse. 

These data were compared and discussed with several other cases in which different aspects of motion sickness, while 
perhaps not totally independent, are clearly not isomorphic. This independence of function persists generally across 

other situations in which motion sickness is elicited (e.g., after space flight). If so, any independent and uncorrelated 
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aftereffects should be assessed individually as each may have different safety implications. Furthermore, independence 
of function would imply different pathways and perhaps different centers of control. 

In conclusion, we propose that there at least two major pathways involved in cybersickness, each exhibiting different 
performance and safety problems during a motion challenge. We believe that the time course of each pathway needs to 
be systematically measured. Such empirical knowledge may enable us to avoid triggering the ANS pathway by using 
brief bursts of exposure, while simultaneously promoting CNS adaptation during repeated and carefully limited 
exposures. 

REFERENCES 

Baltzley, B. A., Kennedy, R. S., Berbaum, K. S., Lilienthal, M. G., & Gower, D. W. (1989). The time course of 
postflight simulator sickness symptoms. Aviation, Space, and Environmental Medicine , 60( 11), 1043-1048. 
Crosby, T. N., & Kennedy, R. S. (1982). Postural disequilibrium and simulator sickness following flights in a P3- 
C operational flight trainer . Paper presented at the 53rd Annual Aerospace Medical Association . Bal Harbor, FL. 
Dolezal, H. (1982). Living in a world transformed: Perceptual and performatory adaptation to a visual distortion . 
New York: Academic Press. 

Held, R. (1965). Plasticity in sensory-motor systems. Scientific American , 72 , 84-94. 

Hennessy, R. T., Sharkey, T. J., Matsumoto, J. A., & Voorhees, J. W. (1992). Simulator induced alteration of 
head movements (SIAHM). A collection of Technical Papers: AIAA/AHS Flight Simulation Technologies 
Confer ence (pp. 29-36). American Institute of Aeronautics and Astronautics, Inc. (AIAA-92-4134-CP). 

Hettinger, L. J., Kennedy, R. S., & McCauley, M. E. (1990). Motion and human performance. In G.H. Crampton 
(Ed.), Motion and space sickness (pp. 

411-441). Boca Raton, FL: CRC Press. 

Kennedy, R. S., Lane, N. E., Berbaum, K. S., & Lilienthal, M. G. (1993). Simulator Sickness Questionnaire 
(SSQ): A new method for quantifying simulator sickness. International Journal of Aviation Psychology , 3(3), 
203-220. 

Kennedy, R. S., Lane, N. E., Lilienthal, M. G., Berbaum, K. S., & Hettinger, L. J. (1992). Profile analysis of 
simulator sickness symptoms: Application to virtual environment systems. Presence , _1(3), 295-301. 

Lane, N. E., & Kennedy, R. S. (1988). A new method for quantifying simulator sickness: Development and 
application of the simulator sickness questionnaire (SSQ) (Technical Report, EOTR 88-7). Orlando, FL: Essex 
Corporation. 

McCauley, M. E., & Sharkey, T. J. (1992). Cybersickness: Perception of 
self-motion in virtual environments. Presence , j_(l), 311-318. 

Money, K. E. (1970). Motion sickness. Psychological Reviews , 50(1), 1-39. 

Money, K. E., Lackner, J. R., & Cheung, R. S. K. (1996). The autonomic nervous system and motion sickness. In 
B.J. Yates & A.D. Miller (Eds .), Vestibular autonomic regulation (pp. 147-173). New York: CRC Press. 

Oman, C. M. (1991). Sensory conflict in motion sickness: An Observer Theory approach. In S.R. Ellis (Ed.), 
Pictorial communication in virtual and real environments (pp. 362-376). New York: Taylor & Francis. 

Parker, D. E. & Parker, K. L. (1990). Adaptation to the simulated stimulus rearrangement of weightlessness. In 
G.H. Crampton (Ed .), Motion and space sickness (pp. 247-262). Boca Raton, FL: CRC Press. 

Pausch, R., Crea, T., & Conway, M. (1992). A literature survey for virtual environments: Military flight simulator 
visual systems and simulator sickness. Presence , J.(3), 344-363. 

Reason, J. T. (1978). Motion sickness adaptation: A neural mismatch 
model. Journal of the Royal Society of Medicine , 71 , 819-829. 

Reason, J. T., & Brand, J. J. (1975). Motion sickness . London: Academic Press. 

Ungs, T. J. (1987). Simulator sickness: Evidence of long-term effects. Proceedings of the 31st Annual Human 
Factors Society (pp. 505-509). Santa Monica, CA: Human Factors Society. 

von Holst, E. (1954). Relations between the central nervous system and the peripheral organs. British Journal of 
Animal Behavior , 2, 89-94. ~ 

Welch, R. (1978). Perceptual modification: Adapting to altered sensory environments . New York: Academic 
Press. 




11-1 


MOTION SICKNESS SIDE EFFECTS AND AFTEREFFECTS OF 
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MOUNTED VISUAL DISPLAYS 
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ABSTRACT 

We have investigated side effects and aftereffects evoked by moving the head to interact with a virtual environment (VE) 
shown in a helmet mounted visual display (HMD). The graphics computer of such a VE must monitor the HMD’s 
spatial orientation and position in order to present images from the proper perspective. Delays between head movements 
and image updating cause aberrant visual motion of a virtual world. We found that above delays of 40 ms motion 
sickness and postural instability are evoked minutes after head movements begin. The severity of side effects is a 
function of the latency between head movement and visual update delay. Fifteen minutes of making head movements in 
a VE with a 254 ms delay causes motion sickness severe enough to make 28% of subjects withdraw from the situation. 
Users appear recovered 15 minutes after VE exposure ends if they remain immobile, but normal activities quickly revive 
their motion sickness symptoms, indicating that they were sensitized by exposure to visual update delays. We conclude 
that visual update delays are a unique cause of side effects and aftereffects in VEs utilizing HMDs. 

INTRODUCTION 

Virtual environment (VE) technology has many potential applications but current technology leads to side effects and 
aftereffects for users. Motion sickness is an acknowledged problem in VEs (DiZio & Lackner, 1992, 1997; Wilson, 
1996) that needs to be solved because it is aversive and hinders performance of normal activities after the user leaves the 
VE. 

We have attempted to document the incidence and etiological factors of motion sickness side effects and after effects 
associated with a prevalent VE configuration which utilizes an HMD. For experimental investigation, we developed a 
facsimile of the VESUB system developed at NAWC-TSD in Orlando, Florida for training naval candidates to navigate a 
surfaced submarine in a harbor. Subjects view a virtual harbor scene from the perspective of a surfaced submarine. One 
sub-goal of both VESUB and of real officer of the deck training, is for the user to develop spatial awareness by scanning 
the environment for signs and landmarks spread out over 360°. Therefore, the VESUB system and our experimental 
system utilize HMDs, which enable presentation of a panoramic view of the virtual world from a perspective that can be 
adjusted by glancing about in a natural manner. 

The scenery of a virtual world in an HMD is carried around by the head instead of staying fixed like the real scenery. In 
such VEs, head position is monitored by a tracking device and fed to a graphics computer which updates the user’s 
perspective and computes the proper image for the HMD display units. This gives the appearance of a stable virtual 
world and appears to contribute to the sense of immersion in the VE (Deisinger, Cruz-Neira, Riedel, & Symanzik, 1997). 
However, the process is imperfect in practice because of delays in the tracking device, the transmission of tracker signals, 
graphical computation and rendering time and others. These delays cause aberrant slippage of the retinal image when a 
head movement is made. That is, the HMD images commence moving later than the head and continue for a latent period 
after the head movement is complete. 

In flight simulators utilizing large screen displays, motion sickness and postural instability are prevalent side- or 
aftereffects, but head movements per se have never been identified as etiological factors (Kennedy, Hettinger, 
Lilienthal, 1990). In our VE, the first and most obvious thing we noticed was that users rapidly became motion sick 
when they looked about wearing the HMD but did not complain if they kept still while viewing a stationary virtual 
world. The only visual motion in our VE was brought about by delayed updating of the visual scene in the HMD during 
head movements. Thus, we launched an effort to determine whether visual update delays associated with head 
movements are a specific cause of side-effects and aftereffects in VEs utilizing HMDs. 


Paper presented at the RTO HFM Workshop on u The Capability of Virtual Reality to Meet Military 
Requirements ” held in Orlando , USA, 5-9 December 1997, and published in RTO MP-54. 
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VE SYSTEM CHARACTERISTICS 

VE computer and software : The virtual environment we experimented with was generated with a Silicon Graphics 
ONYX Reality Engine II computer (SGI). The virtual world was a harbor scene from the perspective of the sail of a 
surfaced submarine. The boat was still in the water, and there was no wave motion or any other external visual motion. 
The harbor channel was marked by buoys in the water and range markers on land. Geographic features, clouds and the 
water surface provided ample visual contrast. The HMD model we used was a LEEP Cyberface II. The LEEP has LCD 
displays 479 pixels wide by 234 high and uses optics which preferentially magnify the periphery of each LCD image. 
We used a psychophysical technique to determine that the LEEP FOV is 128° wide by 74° high, with 30° of binocular 
overlap. We calculated an angular resolution per pixel of 9.8 arcmin in the center graduated to 19.5 arcmin at the edges. 
In all our experiments, the rate of computing video frames was 30 Hz. The devices we used to track HMD position were 
a Polhemus 3SPACE FASTRAK magnetic device with a 19.2K baud serial interface or a custom-made mechanical 
device connected directly to the VME bus via an A/D board. We measured the end-to-end visual update delay of the 
system with both tracking devices. A PC computer recorded signals from photometric devices picking up activity 
directly from the HMD display units and angular rate sensors on the HMD shell, while the VE was running on the SGI. 
The minimum delay we could achieve between onset of HMD movement and image movement on the HMD displays 
was 67 ms with the Polhemus and 21 ms with the mechanical device. We purposely added delays to the VE with a 
software ring buffer in order to achieve an adequate range of experimental visual update delays. 

EXPERIMENTAL PARADIGM 

Two separate experiments were conducted. The first experiment assessed motion sickness severity at visual update 
delays of 67, 159, 254 and 355 ms utilizing the Polhemus tracking device. Twenty one subjects participated in all 
conditions. Seven new subjects were recruited for the second experiment in which motion sickness severity was assessed 
at update delays of 21, 39, 80 and 163 ms, achieved with the mechanical tracker. 

Experimental exposure to the VE lasted 15 minutes, split into five 2 minute sequences of head movements divided by 1 
minute periods for rest and recording of motion sickness symptoms. In the 2 minute sequences, a pre-recorded audio 
tape announced landmarks in the VE for the subject to look at every five seconds (24 total movements). The required 
head movements ranged from 12° to 180° amplitude in the horizontal plane, between 25° up and 15° down in pitch. 
Subjects had been shown an aerial view of the virtual harbor before donning the HMD in order to familiarize them with 
the landmarks. The subjects stood while doing the task and most of the HMD's weight was supported by long elastic 
cords. Typically, subjects took less than one second to turn to a new target, using a combination of eye, head and torso 
rotation and sometimes shifting their feet, and then they stood still until the next landmark was called. 

Acute symptoms of motion sickness were evaluated according to the criteria of Graybiel, Miller, Wood & Cramer 
(1968). The checklist derived from this scale measures five cardinal signs and symptoms of motion sickness, including 
nausea, pallor, sweating, salivation and drowsiness, and additional qualifying symptoms such as headache, dizziness and 
eye strain. Scores in the 8-15 point range indicate severe malaise. The checklist was filled out after each two minute 
sequence of head movements in the VE and 15 minutes after leaving the VE. 

MOTION SICKNESS SIDE EFFECTS IN THE VE 

Figure 1 illustrates the most severe motion sickness recorded during the 15 minute VE exposure period, for both 
experiments. The severity of symptoms increases monotonically as a function of the visual update delay. Analysis of 
variance indicated significant effects of visual update delay: F(3, 60)= 10.43, p<.0001 in Experiment 1 (67, 159, 254, 355 
ms delays) and F(3, 18)= 7.11, p<.0067 in Experiment 2 (21, 39, 80, 163 ms delays). The severity of symptoms is 
significantly greater than zero at every delay except 19 ms (p<.03 at least, individual t tests). The ratings register from 
mild malaise on the Graybiel scale at 39 ms to moderately severe malaise at 355 ms. The severity of motion sickness 
seems to asymptote between 254 and 355 ms (no significant difference between these two). In the 254 ms delay 
condition, 6 of 21 of subjects (28%) refused to complete the full 15 minute exposure because they were too close to 
vomiting. The full spectrum of symptoms was reported in all conditions. 
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15 min HM + 15 min Rest-1- 2 min HM + 2 min HM 

w/HMD w/o HMD w/o HMD 


Figure 1. Motion sickness elicited by voluntary head movements (HM) in the initial 15 minute exposure to a VE with a 
254 ms delay in update of the HMD image, after 15 minutes of rest without the HMD, 2 minutes of HM in a natural 
environment and 2 minutes in the VE again. 

| Polhemus Tracker, N=21 



Figure 2. Motion sickness severity as a function of visual update delay, in VEs created with two different tracking 
devices. The plot shows the maximum severity rating evoked during 15 minutes of intermittent voluntary head 
movements. 

MOTION SICKNESS AFTEREFFECTS 

A preliminary analysis of the first 8 subjects indicated that motion sickness symptoms had abated almost entirely in 15 
minutes after the VE exposure period was over and the subjects had been resting quietly. Therefore, we asked the 
remaining subjects after the 15 minute rest period to perform two additional head movement sequences, guided by the 
same 2 minute tape recorded directions as before. 

In the first post-rest sequence, the HMD was not worn and the location of VE landmarks was marked by tags placed on 
corresponding locations in the real laboratory environment. As soon as the motion sickness symptoms were recorded, the 
subjects donned the HMD again and made another 2 minute sequence of head movements in the VE and reported 
symptoms a final time. The extra conditions were only done after the 254 ms delay VE exposure. Figure 2 illustrates the 
complete pattern of results for the 13 subjects who completed all conditions. 

The motion sickness severity experienced by this sub-sample during 15 minutes in the VE was 5.7 points, which is 
similar to the value of 6.2 for the whole sample. Fifteen minutes after leaving the VE and resting in a normal 
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environment the malaise level was only 1.0, not significantly above zero. Subsequently performing head movements in a 
natural environment for 2 minutes brought the motion sickness score back up to 4.4 points, which was significantly 
above 1.0. Returning to the VE and making head movements for 2 minutes brought the symptom level up above the 
level of the original fifteen minute exposure, to 6.7points. 

CONCLUSIONS 

The results indicate that motion sickness is a serious potential side effect and aftereffect of using a HMD as a visual 
interface to interact with a VE. Motion sickness severe enough to convince subjects they might vomit can be elicited 
within minutes of exposure to the VE. Users experience no motion sickness unless they move their head to scan the 
environment. In the VEs we studied, there was no simulated motion of the user’s base of support or of other objects. 
The only visual motion was generated by voluntary head movements. Thus, head movements are a sufficient cause of 
motion sickness in VEs incorporating HMDs. This is not the case in flight simulator. 

Overt motion sickness decays quickly after a 15-minute VE exposure is over, but users remains sensitized. Anecdotal 
reports indicate the period of sensitization is much longer than the 15-minute period in which we have done formal 
evaluations; up to hours is possible. During the sensitized period, even normal activities can bring the symptoms back, 
and returning to the VE escalates symptoms immediately to the prior maximum level or above. 

Lags between head movements and compensation of the image in the HMD are a powerful etiological factor in this form 
of motion sickness. Such lags distort visual motion of the virtual world during head movements. The motion is distorted 
in the sense that the normal coordination of the vestibulo-ocular and optokinetic reflexes does not result in stabilization 
of the retinal image. The magnitude of aberrant visual motion and the severity of motion sickness both increase in 
proportion to the visual update latency. A latency of 19 ms is below threshold for eliciting motion sickness, under the 
conditions we tested. 

These findings can aid in ameliorating or circumventing motion sickness in VEs. They suggest that careful 
measurements of end-to-end visual update delay of VE systems is necessary to anticipate and circumvent side effects. 
To avoid motion sickness, users can limit their head movements, VE designers and engineers can strive for sub-threshold 
visual update delays, or the power of human sensorimotor adaptation can be exploited. 

We are currently attempting to define other etiological factors and side effects that must be take into consideration. Our 
results indicate that postural instability is another serious side effect/aftereffect and that the field of view, weight and 
spatial resolution of an HMD as well as the computation rate of video frames are other etiological factors. 
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HUMAN BODY MODELING AS A HUMAN FACTORS 
ENGINEERING TOOL 

Nick Beagley 

Centre for Human Sciences, DERA 
Famborough, Hampshire GUM OLX 
United Kingdom 

Computer based human body modeling provides a tool with which human factors issues can be assessed early in a 
system's design. By populating an interactive visualisation of a design with representative mannequins issues such as fit, 
reach and vision can be evaluated and design changes recommended at the concept stage. Body modeling tools are 
continually advancing, offering increasingly sophisticated dynamic representations of the human operator. For the 
influence of these tools to spread further, they must minimise the demands placed on their users. The user requires a 
consistent, stable tool which answers the traditional questions of human factors evaluation at an early point in the design 
cycle. 

INTRODUCTION 

The task of the human factors engineer is to match the design of systems to the requirements and capabilities of their 
human users. Effective equipment design must balance a range of constraints whilst seeking to meet the product's goals. 
A growing engineering approach has been to use computer aided design (CAD) as a means of testing alternative 
solutions to the design problem. For usability to be considered during product development it is necessary to represent 
the capabilities and constraints of a system's human component within this design process. Human body modeling can 
support this requirement by providing the human factors engineer with a tool that is compatible with the CAD focused 
design methodologies of product developers. 

Virtual reality (VR) now allows dynamic mannequins to be interactively fitted and manipulated within three dimensional 
models of system concepts. This offers a means of assessing and presenting ergonomic issues at an earlier point in the 
product lifecycle than previously possible. Identifying and correcting design conflicts at this early stage offers savings in 
time and money. The process of product design and specification can often be faced with user uncertainty, the instability 
of user requirements, and a communication gap between the designers and the users. Computer modeling starts to 
address these problems by allowing user evaluation of the virtual prototype as the basis of defining and refining a 
common concept of the required system. 

There is a widespread perception of VR in which the user puts on a headset to enter an alternate reality. This bears little 
relation to the current use of body modeling for human factors evaluation. Although many of the tools available will 
output to a stereo headset, these systems tend to be viewed through a desktop screen. Whilst immersive evaluation of a 
design promises to be a powerful medium for considering the user's perspective, the benefits of immersion do not, as yet, 
offset the cost and limitations of its supporting technology. Despite this, the emergence of desktop VR tools has raised 
the potential for applying novel techniques to human factors evaluation and input to design. Whilst the immersive 
interface promises another level of model experience, the true value of these environments centres on their ability to 
visualise and interact with a populated concept model, for which desktop presentation is adequate. Immersive displays do 
not currently offer an advantage for the core human factors tasks of analysing fit, reach and vision for a given population. 

THE HUMAN MODEL 

Modeling the human remains a major challenge to the development of virtual worlds. Whilst inanimate objects such as 
buildings, machines and landscapes have been convincingly simulated for many years the representation of organic 
forms, particularly the human, has presented a greater obstacle. The inherent complexity and the absence of uniformity 
associated with organic forms has made them difficult to simulate both visually and in terms of their behaviour. 

Synthetic environments have advanced to provide detailed interactive worlds in which simulated forces can interact over 
a distributed network. Representation of the human within these environments have tended to either group the human 


Paper presented at the RTO HFM Workshop on u The Capability of Virtual Reality to Meet Military 
Requirements ”, held in Orlando, USA, 5-9 December 1997, and published in RTO MP-54. 
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within an aggregate force or relegate the user to a simplified component of a more easily simulated entity such as a tank, 
a plane or a ship. 

Recognition of the influence of human behaviour and capability on the real world activities on which synthetic 
environments have been modelled has led to work to improve the representation of the human. 

The Centre for Human Sciences’ (CHS) Integrated Performance Modeling Environment (IPME) project seeks to shape 
the performance of synthetic environment entities based on an integrated set of human performance models. Other work 
within DERA is looking to incoiporate human performance factors within the emerging command agent technology. 
Great improvements have been made to the visualisation of dismounted infantry within SEs with entity models such as 
DI-Guy (Boston Dynamics). Whilst the domains of human modeling for SEs and body modeling for workplace analysis 
are largely separate, they are each working to enhance the human model and may in the future meet in the middle. 

The functional complexity of the body model required for workspace assessment depends on the aims of the evaluation. 
At its most basic, a line of sight assessment could be made by positioning your views of the environment at the 
operational eye heights for your range of users. Whilst offering an initial assessment, this overlooks the figure’s dynamic 
interaction with the environment such as sitting in a chair or turning to see a display. By using a fully articulated human 
figure it is possible to make a more complete evaluation of the environment taking into account access, fit, reach, vision, 
etc. within a system model in a way which relates to the methods and design principles regularly practised in the 
ergonomic evaluation of hardware prototypes and completed systems. 

COMMERCIAL BODY MODELING TOOLS 

A wide range of commercial body modeling tools are available. At the lowest level, it is possible to buy a selection of 
static mannequins to populate architectural models. Whilst these figures are intended for presentation purposes they 
might be considered to aid the evaluation of a design by illustrating an area's intended usage. At the other extreme 
advanced biomechanics models are available which allow the user to develop and analyse musculoskeletal models 
(Musculographics, Inc.). Between these extremes there are a number of tools which offer dynamic body models for the 
purpose of design evaluation. An outline of some of these tools is provided in Table 1. 


Android 

(Mechanical Dynamics) A graphical human body model for 
the ADAMS CAD system. The software enables the user to 
create human models to study kinematic, static, and dynamic 
behaviour. 

COMBIMAN 

(CSERIAC) An interactive CAD model offering body size 
and proportions for Air Force and Army pilots. Provides a 
tool for reach, strength and vision analyses. Human model is 
based on a 35 segment representation of the human skeletal 
system. 

Crew Chief 

(CSERIAC) A derivative of COMBIMAN which provides 
an interactive CAD model of an aircraft maintenance 
technician. The model takes into account body size, posture, 
clothing, strength and vision for evaluating designs for their 
ease of maintenance. 

ERGO 

(Deneb Robotics, Inc.) A module of the Deneb CAD 
modeling toolset which offers rapid prototyping of human 
motion within the work area. The model incorporates 
forward and inverse kinematics for limb manipulation. 

JACK 

(Transom) Jack is a high fidelity human model which is able 
to interact with the virtual environment. Features include 
walking, grasping, eye tracking, animation, collision 
detection and dynamic strength. The environment provides 
advanced visualisation features such as lighting and mirrors. 

Kinemation 

(Alias Wavefront) Three dimensional character animation 
through forward and inverse control of the skeleton. Includes 
a behavioural model for its skin model linked to time or the 
skeletal position. 
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MannequinPRO 

(HumanCAD) A development tool to create human 
mannequins for import to other CAD packages such as 
AutoCAD. Allows user to create a range of body size 
characteristic and postures, outputting a static model. 

MDHMS > 

(Boeing) The McDonnell Douglas Human Modeling System 
(MDHMS) is an in-house body modeling tool which 
includes a three dimensional animated human mannequin. 

The system provides articulated limbs and inverse 
kinematics for the simulation of assembly, operations and 
maintenance. 

Poser 

(Fractal Design) A body modeling tool offering a range of 
body sizes and body types. Although intended as a 
visualisation tool, it supports interactive body posture and 
limb control and allows the output of static CAD models. 

Safework 

(Genicom) A detailed human model whose features include 
animation, vision analysis, collision detection, and posture 
analysis leading to a comfort score. The model allows user 
alteration of critical variables, adjusting the other 
anthropometric variables based on statistical variations. 

Sammie 

(British Technology Group) A body modeling environment 
offering anthropometric and morphological control for the 
analysis of fit, reach, vision and posture. 


Table 1. Commercial Body Modeling Tools 


Computer Aided Workplace Design at the CHS 

Computer aided design at the CHS started with the acquisition of the Sammie software tool in the late ‘80s. The Sammie 
system provided the vehicle ergonomics group with a virtual environment in which an articulated figure could be scaled 
to represent key anthropometric measurements. The mannequin could then be placed within a skeletal model of a 
proposed operating environment such as a vehicle crewstation. Figure 1 shows the Sammie mannequin positioned within 
a concept crewstation which was designed and evaluated by the group using this tool prior to the construction of an 
experimental crewstation. 

After mastering the precise command set and syntax of its UNIX operating system, necessary to run and maintain the 
software, the user had to overcome a cumbersome interface to manipulate the mannequin within the virtual environment. 
The system was full of promise, offering a method of viewing and evaluating a concept environment from the 
perspective of a representative user. However, the time and effort required to set up a populated environment for 
evaluation reduced the number of occasions on which computer modeling could be justified over paper based techniques 
and hardware mock-ups. 



Figure 1. Sammie Mannequin within a Concept Crewstation Model 
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Despite the limitations of the group's early body modeling tools, the recognised potential for improved input to the 
product design cycle led to their subsequent acquisition of the JACK body modeling tool. Improvements in affordable 
computing power, JACK'S enhanced anthropometric model, its more usable interface and its advanced functionality were 
considered to counter many of the limitations of earlier systems. Continual improvements to the modeling tool and to the 
power of its host computer platforms have led to a gradual rise in its application to specific human factors design 
projects. 

The group has explored the extent to which the body modeling tool can be usefully applied. Designs evaluated using this 
tool include a hand held imaging device, the control panel for a remotely piloted vehicle, office environments, 
crewstation designs, and a harbour traffic control room. 

Figure 2 shows a virtual world generated in the body modeling environment to assess a control room. At one level, 
design alternatives to the control room were assessed against considerations such as furniture design, inter-operator 
communication and the position of display screens. In addition, it was important to consider the controllers' primary 
display, the room’s windows. A simplified model of the harbour was generated in addition to the redesigned contiol 
room supporting an evaluation of the design's influence on the visibility of the shipping lanes. 



Figure 2. JACK Model of a Harbour Control Room Including External View 


The utility of body modeling tools is not limited to design evaluation against human factors guidelines. Figure 3 
illustrates a detailed model of a closed circuit television control room modelled and evaluated by CHS. Interactive 
control and animation of a human mannequin within this environment supported a standard analysis of fit, reach and 
vision within the proposed room. Techniques such as texture mapping and shadowing helped to quickly build a realistic 
representation of the environment. By visualising the environment, the model offered an accessible medium for the 
stakeholders to develop a common design concept. The model was used by representatives of the designers and the users 
to review the environment and interactively make changes to its proposed design. At the same time, the knock-on effects 
of these changes were assessed from a human factors perspective. This process led to the gradual evolution of the design 
as alternatives were prototyped and iterated. 
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Figure 3. Close Circuit Television Control Room 


USER REQUIREMENTS FOR BODY MODELING TOOLS 

The cases described illustrate real applications of body modeling as a tool to provide human factors input within the 
design process. Computer based body modeling is a relatively recent technique and there is considerable scope for 
improvement to the tools. The value of this technique has provided the motivation to work within the shortcomings of its 
supporting tools. There remain many reliability, usability and functionality issues relating to the tools applied within 
CHS which currently inhibit their wider use. 

Reliability 

Workspace evaluation requires the body modeling tool to be consistent, stable and valid. The user needs to be confident 
in the tool’s ability to repeatedly provide an accurate interactive model of the human and its environment. Human body 
models have tended to increase in complexity in order to provide a more accurate representation of the individual. 
However, the multiple degrees of freedom which characterise the joints of the human body place significant processing 
demands on the software simulation. When interactively manipulating a mannequin within a 3D model, these models 
must balance the constraints of the mannequin’s skeleton with the constraints of the environment and the instructions of 
the tool’s user. There is a risk that small variations in position and action will lead to a wide variation in mannequin 
response. This can reduce the apparent realism in movement of the human figure. In addition, these inconsistencies 
challenge the use of these tools for workspace evaluation which relies on a degree of repeatability before conclusions can 
be drawn concerning fit and reach. Furthermore, the increase in the number of calculations performed during simulation 
raises the risk of software failure. The effort invested in developing an environment and an associated evaluation 
procedure can be quickly lost in a software crash. The effect of this can be a loss of confidence in the tool leading to a 
more narrow application of the tool’s advertised capabilities for future projects. 

USABILITY 

The usability of these tools has been greatly enhanced by the improving cost to power ratio of computer hardware. It is 
now possible to display and interact with fully shaded 3D models. Past human modeling relied on wire frame 
representation of the environment which was often difficult to interpret. Despite these improvements, the overall 
usability of these tools tend to fall short of their potential. An apparent side effect of increased processing power has been 
an increase in the overt functionality of these tools. In some cases the user is faced with a range of functions, each 
requiring the definition of numerous obscure attributes before a mannequin behaviour can be implemented. Whilst 
increasing control over the simulated human is a positive goal, the manner of its implementation is critical. It should not 
be necessary to alter balance controls in order to manoeuvre a mannequin onto a seat. These tools need to automate fine 
control of the mannequin with the option of a manual override by the user at any point or level of detail. 
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Control within 3D virtual environments remains a difficult problem to solve. Most systems allow mouse control, 
modulated by holding down combinations of the mouse or keyboard keys to control movement in each axis. Other 
devices such as joysticks and space balls offer an alternative without offering a truly intuitive interface. 3D position 
sensors such as the polhemus system (Polhemus) allow the mannequin to mimic the limb movements of the user but 
remain an expensive and cumbersome solution. A further alternative is provided by an instrumented physical mannequin 
called the “Monkey” (Digital Image Design Inc.) which, when linked to the body modeling tool, allows direct 
manipulation of the human model. 

Despite these control options, human body modeling remains a cumbersome tool. When evaluating a design, certain 
questions are asked such as, can the population reach control A?, or can the population see screen B? For these tools to 
reach their potential, they must provide answers to these questions with the minimum of effort on the part of the user. 
One approach to achieving this is to provide the mannequin with a working understanding of the environment. For 
example, it should know what a chair is and how to sit on it. It should know what a screen is for, how to turn it on and 
whether it is in a comfortable position. What is needed is for the computer to take on the labour intensive tasks of 
mannequin control to free up human effort to consider the information provided by the tool. 

Functionality 

Whilst expanding functionality has been highlighted as a potential hindrance to usability, enhancement of the human 
body model could help to improve tool usability and user confidence, particularly where added complexity is hidden 
from the user. Continuing efforts to improve the reliability and consistency of these tools require the improvement of 
existing functions such as collision detection and the development of new functions such as a method of arbitrating 
between the numerous inputs to the model to achieve consistency in the mannequin’s actions. All of these advances will 
be diminished if they are not implemented in way that is sensitive to the users’ requirements. 

Work to enhance the model of human performance and behaviour within synthetic environments is one area in which the 
human body model could be developed into a more powerful human factors tool. For example, current evaluations 
require the expert to make a judgement of the user’s ability to operate in an environment for a prolonged period. By 
including and improving human behaviour and performance models for mannequins it may become possible to define 
operational scenarios to test variables such as temperature, fatigue and stress when considering a proposed layout. 
Models of attention and information processing capacity could be used to highlight features of a design which might lead 
to an insupportable load under situations of high activity or a likelihood of error during periods of low activity. 

Although initially rejected by this discussion, the potential contribution of immersive interaction within the virtual body 
modeling environment may provide new insights to design once the technology has matured sufficiently. The ability to 
experience a proposed environment from the perspective of a 97th percentile maintenance technician or even through the 
eyes of a visually impaired operator has the potential to demonstrate the importance of accounting for variation, both 
human and situational, in design in a new and powerful way. 

CONCLUSION 

Human body modeling provides the human factors engineer with a new tool which ties together the opportunities of 
computer aided design and the user centred techniques of prototyping and iterative design. It was stated at the start of this 
paper that “the task of the human factors engineer is to match the design of systems to the requirements and capabilities 
of their human users.” It seems reasonable to expect this goal to be prominent in the design of the tools offered to support 
this process. Continued efforts to improve these tools along these lines should secure the place of this technology as an 
important tool for supporting a human factors input to design. 
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ABSTRACT 

Immersive simulation techniques such as Virtual Environments (VE) can revolutionize human factors engineering and 
training projects provided that they are carefully validated. Is human performance in the virtual world the same as in the 
real world? When visual aspects perceived on a virtual ship differ from those perceived on a real ship, human factors 
engineering studies may yield non-optimal designs. When interactions with virtual worlds are not natural, training may 
not transfer to the real world. 

I will discuss three studies that compared human task performance in real and virtual (HMD) environments. First, we 
carried out spatial perception experiments and measured localization performance: how well can people indicate the 
center point between two objects in identical virtual and real environments. Second, manipulation performance was 
measured: how well can people grab, turn and position objects in virtual environments and what adaptation effects occur 
when returning to the real world. Third, we compared the assessment of ergonomic aspects of identical virtual and real 
ship bridges. 

Discrepancies found between the results for the real and the virtual bridge are discussed in terms of challenges with 
respect to the quality of head-mounted display optics and tracking devices and, most importantly, with respect to natural 
interfaces needed for manipulation (virtual hand control) and for moving around in virtual worlds (intuitive navigation 
methods). 

SPATIAL PERCEPTION IN VE 

This study focused on the quantification of spatial perception in virtual environments as a quality measure for realism 
and functionality. The accuracy of spatial positioning tasks was psychophysically quantified for subjects in an 
immersive visually simulated environment: a 5m x 5m Engine Control Center of a Dutch M-frigate. Performance in the 
real environment was compared with the performance of identical tasks in its virtual counterpart. 

One type of task (bisection) was to position a hand-held pointer at an imaginary bisection point between two markers in 
3D space (separated 2m to 6 m). Another type of task (positioning) was to position the pointer at a certain distance from 
the wall. Virtual space was simulated using a Polhemus' Fastrak head-tracking device, Evans and Sutherland’s ESIG 
2000 high performance image generator (64 ms delay time) and an Eyegen 3 head-mounted display (colour, stereo, a 40 
degree field of view). 

The typical distance between indicated and actual bisection points was approximately 20 cm in the real versus 30 cm in 
the virtual environment (a 50% increase in errors). Positioning tasks show an even higher increase in errors 
(approximately 90%). These data show that our 3D spatial bisection and positioning tasks are sensitive to the impairment 
of spatial perception in (high performance) virtual environments due to a simultaneous decrease of spatial resolution, 
field of view and/or delayed visual feedback. In a separate experiment it was shown that field of view alone was not 
responsible for the degraded performance in VE. Further, experiments in a virtual environment with only isolated 
markers in an empty environment show that subjects integrate environmental structure spatially for optimal positioning 
performance. The detailed modelling of environmental structure appears to be an important property of virtual (and 
probably of real) environments. 
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MANIPULATION PERFORMANCE IN VE 

The application of Virtual Environment techiniques in training projects often requires a high-level natural interaction 
with the environment. Virtual hand control is the most natural manual interaction method called direct manipulation. 
Direct manipulation allows users to grasp, rotate and move virtual objects with a realistic virtual hand that mimics the 
real hand. Such intuitive manipulation methods potentially enable advanced applications in the field of interactive design, 
training, medicine, etc. However, they should be carefully evaluated. 

We have studied manipulation performance in virtual environments using direct (virtual hand control) manuplation 
compared with indirect (mouse driven 3D cursors) manipulation. These manipulation methods were tested under 
monoscopic and stereoscopic viewing conditions. Participants were asked to discriminate, grasp, pitch, roll and position 
virtual objects. Both speed and accuracy of manipulation tasks were measured. 

Virtual hand control proved to be significantly faster (35%) and more accurate (40%) than 3D mouse cursor control. 
Virtual hand control affected more head movement which was available in both conditions. Head movements induce 
motion parallax (a powerful visual depth cue). Further, it was shown that the speed and accuracy of manipulations 
strongly improve under stereoscopic viewing conditions. Stereoscopic viewing also significantly increased the sensitivity 
to size differences between objects (50% better). 

ADAPTATION OF EYE-HAND COORDINATION IN VE 

Above we mentioned that performance (speed, accuracy) of direct manipulation (virtual hand control) is much faster and 
far more accurate than that of indirect manipulation (traditional mouse driven 3D cursor control). However, in current 
virtual environments the virtual hand may not always be exactly aligned with the real hand due to imperfect optics or due 
to tracking errors. Such misalignment may cause an adaptation of the users' eye-hand coordination and possibly a 
decrease in manipulation performance compared to aligned conditions. Therefore, we have studied visuo-motor 
adaptation as well as performance decrease under misaligned virtual hand conditions using a prism adaptation paradigm. 

Participants were immersed in an interactive virtual environment with a deliberately misaligned virtual hand position (a 
lateral shift of 10 cm). We carried out pointing tests with a non-visible hand in the real world before (pre-test) and after 
(post-test) immersion in the virtual world. A comparison of pre- and post-tests revealed after-effects of the adaptation of 
eye-hand coordination in the oppositie direction of the lateral shift (negative after-effects). The magnitiude of the after¬ 
effect was 20% of the lateral shift under stereoscopic viewing conditions. Interestingly, decreased manipulation 
performance (speed/accuracy) during the immersion with misaligned hand compared to aligned hand conditions was not 
found. The occurrence of negative after-effects in lateral direction indicates lower level parameter adjustment of eye- 
hand coordination. This is promising for those interested in using virtual hands to acquire visuo-motor skills. Acquired 
skills in VE are likely to transfer to the real world. 

DESIGN TASKS 

By order of the Royal Netherlands Navy, TNO developed and evaluated a prototype VE-system which enables the 
interactive 3D design of operational areas of a ship and which offers a wide range of specific human factors engineering 
tools. The evaluation of the prototype system focussed on the functionality and validity of the system. We have 
evaluated the VE-system on the basis of a series of tests carried out by eight subjects on the virtual bridge of the Hr.Ms. 
Mercuur. For verification purposes, the results of the tests on the virtual bridge were compared with those obtained with 
identical tasks on the real bridge. First, we tested reachability for examining the perceptual and geometrical quality of 
the proto-type system. Second, accessibility was tested to quantify the speed and ease of navigation and obstacle 
avoiding in VE. Third, we tested how well participants could manually grasp and position a bearing-compass given a set 
of ergonomic constraints. 

Results showed serious perceptual deformations of spatial dimensions in the virtual environment for all tests. 
Misperceptions may be partly explained by imperfect head-mounted display (HMD) optics. For the other part, modeling 
and tracking problems must have caused the problems. Further, objects in the virtual environment (VE) were not always 
recognizable and in some cases appear to be (or are) different from the real environment (RE). This caused serious 
discrepancies between the results of our design task in VE and RE. 
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We also found that accessibility studies in VE take significantly longer than in RE and that subjects collide with 
obstacles in VE far more often in VE than in RE. This finding is likely to be due to the relatively narrow field of view of 
current HMDs and to non-intuitive navigation methods. 

These results indicate that virtual environments have to be very carefully modeled and in sufficient detail before they 
can be used for evaluation tasks such as reachability and accessibility tests. Furthermore, head-mounted displays have to 
be carefully tuned to the properties of the human visual system using high quality optics. Feedback delays should be 
reduced. Finally, new, more intuitive methods for navigating in VE may improve a user's sense of orientation in VE and 
the functionality of accessibility studies. 

Knowledge of flaws (as emerging from the current validation studies) enables us to correct VE models and display 
techniques such that human factors studies can be safely carried out in VE. When thoroughly validated, VE can be a 
powerful tool for even more advanced human factors engineering studies (reachability, accessibility, interactive design) 
than the ones that have already been carried out successfully with VE (view, layout). The benefits of VE in terms of 
participation, risk reduction, cost reduction and design optimization are evident. 
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INTERFACE ISSUES IN THE USE OF VIRTUAL ENVIRONMENTS FOR 
DISMOUNTED SOLDIER TRAINING 2 ’ 3 

Bruce W. Knerr, Ph.D. 
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12350 Research Parkway 
Orlando, FL 32826, USA 


ABSTRACT 

In 1992 the U.S. Army Research Institute for the Behavioral and Social Sciences (ARI) initiated a program of in-house 
experimentation to investigate the use of virtual environments (VE) technology to train dismounted soldiers. Since that 
time, we have conducted thirteen experiments examining human performance in VE, training effectiveness and 
transfer of skills acquired in VE to the real world, and side-effects and after-effects of exposure to VE. The tasks used 
have included distance estimation, tracking, object manipulation, visual search, route learning in buildings, building 
search, and land navigation. This paper summarizes results from these experiments related to visual display 
characteristics and methods of locomotion. 

The most common VE display systems, low- to moderate-cost head mounted displays (HMDs), limit performance with 
low resolution and small fields of view (FOVs). Performance on a variety of distance estimation tasks is significantly 
worse than performance on similar tasks in the real world. Providing stereoscopic view improves performance, but 
only at short distances. Increasing the field of view while holding resolution constant improves performance. Linking 
the viewpoint to head movements improves distance estimates and, under some conditions, spatial knowledge 
acquisition. For some tasks, performance using a monitor is better than performance using an HMD, while on other 
tasks, the reverse is true. 

A variety of methods have been used to simulate walking in VE: joystick, spaceball, treadmills, and walking in place 
(with instrumentation to sense steps). Few direct comparisons of these methods have been made. For some tasks, a 
joystick combined with auditory cueing may provide an effective substitute for high-cost locomotion simulators. 

INTRODUCTION 

In 1992 the U.S. Army Research Institute for the Behavioral and Social Sciences (ARI) initiated a program of in-house 
experimentation to investigate the use of virtual environments (VE) technology to train dismounted soldiers. Since that 
time, we have conducted thirteen experiments examining human performance in VE, training effectiveness and 
transfer of skills acquired in VE to the real world, and side-effects and after-effects of exposure to VE. The tasks used 
have included distance estimation, tracking, object manipulation, visual search, route learning in buildings, building 
search, and land navigation. We have now reached the point in our program where it is necessary to synthesize what 
we and other researchers have found, draw conclusions, and make recommendations for the use of VE for Army 
training. We are now beginning that that process. While our primary interest is in the use of VE for training, it has 
been clear from the start that it would be necessary to pay particular attention to the human computer interface. This 
paper will present some preliminary conclusions about two types of interface devices: relatively inexpensive visual 
display devices and devices for the simulation of locomotion. 

The original program objective was to improve the Army’s capability to provide effective, low-cost training and 
rehearsal for Special Operations Forces and Dismounted Infantry through the use of virtual environments technology. 
Subgoals were to: focus on the requirements for individual soldier and leader accomplishment of unit tasks; determine 
the necessary characteristics of VE technology, including fidelity requirements; and evaluate transfer of training and 
performance to the real world. While we have consistently pursued these objectives over the life of the program, 
we have found them to be more complex than originally anticipated. 

We describe our research program as a progression up the levels of a pyramid. Following an initial analysis of the task 
requirements for dismounted soldier training, and a review of the previous research in the use of VE for training (a 
very sparse area when we began), we conducted four experiments to investigate interface effects on the capabilities of 
participants to perform simple tasks in VE. Variables investigated included the type of control device, amount of 
practice on the tasks, stereoscopic vs. monoscopic helmet-mounted displays, and type of display device (monitor, 
boom, or helmet-mounted display). At the next level, we conducted two experiments that addressed the effectiveness 
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of VE for teaching the configuration of and routes through large buildings, and the transfer of the knowledge acquired 
to the real world. Taken together, these results led to an originally unplanned program of research into the 
investigation of distance estimation in VE. At the third level, we investigated the use of VE to represent exterior 
terrain, both for training land navigation skills (identifying landmarks and learning routes), and assessing threats. 
Research at the top of the pyramid (we have combined the top two levels) is investigating the use of VE for training 
team tasks. This experimental series is just beginning. 


Real W orld 


Boom 2C 
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Monitor 



Figure 1. The relative sizes of the fields of view of 
various VE display devices 

This paper will address two sets of interface devices: visual display devices and locomotion simulators: 

There are four major characteristics of HMDs that seem particularly important: 

• Resolution 

• Field of View (FOV) 

• Stereoscopic Vs. Monoscopic Vision 

• Presence or absence of Head Coupling. 

HMDs that we have used include the FakeSpace Labs BOOM2C, the Virtual Research Flight Helmet, and the Virtual 
Research VR4. HMDs differ in their use of color (the BOOM2C uses only two colors, not three), their resolution, and 
their FOV. Figure 1 shows the proportional size of the field of view of these HMDs relative to the normal human FOV 
and a 20” monitor viewed at a distance of approximately 24”. Note that use of these devices restricts peripheral vision. 
The resolution of the displays is poor. Administration of an acuity test in VE has resulted in mean Snellen acuities of 
20/500 and 20/860 for different versions of the VR Flight Helmet, and 20/200 for the BOOM2C (Lampton, Knerr, 
Goldberg, Bliss, Moshell, and Blau, 1994; Lampton, Gildea, McDonald, and Kolasinski, 1996). 



Judged Distances (ft) 

Figure 2. Percent error in distance estimates as a 
function of display device 
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VE DISPLAYS AND DISTANCE ESTIMATION 

To determine the impact of these factors on performance in a realistic task, we developed a distance estimation task in 
which the human participant estimates the distance to a human figure as it moves toward them from a known distance 
of 40 feet. We did this with a number of different display devices in VE (using a virtual image of the human figure) 
and in the real world with a live human. Figure 2 combines the data from several different experiments (Lampton, 
Knerr, Goldberg, Bliss, Moshell, and Blau, 1994; Lampton, Gildea, McDonald, and Kolasinski, 1996; Singer, Ehrlich, 
Cinq-Mars, and Papin, 1995). Errors in the VE are in the direction of underestimating the distance to the figure (i.e., 
reporting it as being closer than it “actually” is). This seems to be consistent across all of the distance estimation tasks 
that we have used, with one exception to be described later. Significant differences among display devices are found 
more-frequently at the shorter distances. Performance generally improves with the increases in the field of view and 
resolution. Stereoscopic presentation improves performance only at distances shorter than 10 feet. 



Figure 3. Percent error in distance estimates as 
a function of head coupling 

As shown in Figure 3 head-coupling, or linking the field of view to head movements, also improves performance at the 
shorter distances (Singer, Ehrlich, Cinq-Mars, and Papin, 1995). 

In order to examine the factors affecting distance estimation in VE more closely, and perhaps develop ways to improve 
distance estimation, Witmer and Kline (in press) used several other tasks. Figure 4 shows the results of an experiment 
in which participants were asked to estimate the distances to cylinders placed in a corridor at distances from 10 to 110 
feet away. They underestimated the distances in both the real and virtual worlds, but underestimated them less in the 
real world. 



Distance to Cylinder (ft) 


Figure 4. Distance estimates to a stationary 
object in a VE and the real world 
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Figure 5. Distance traveled as a function of 
target distance 


To what extent are these results due to participants’ inability to provide good numeric distance estimates in any 
environment? Witmer and Sadowski (in preparation) used a different approach that asked participants to walk 
blindfolded to a target that they had just seen, instead of reporting the distance verbally. Participants in the real world 
condition viewed the targets, then walked the distance in a hallway. In the VE condition, they used a treadmill to walk 
a simulated hallway. Again, the participants underestimated the distances to the targets in both the VE and the real 
world (Figure 5). 

They underestimated them more (i.e., were less accurate) in the VE. We can be confident, therefore, that these errors 
in distance estimation are not some artifact of the use of verbal reports of distance. However, it is interesting to note 
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that the overall level of performance was better in this experiment than in the one described immediately before. 
Distance traversed in VE was about 85% of the true distance, while the verbal reports of stationary observers in the 
previous experiment were about 50% of the true distance. 

It is difficult to examine the effects of the display device characteristics on human performance because we are limited 
to the characteristics of existing displays, which tend to differ on multiple dimensions, not just a single one. However, 
Kline and Witmer (1996) were able to use a software mask to create a smaller FOV for the BOOM2C, without 
changing any other characteristics. In this experiment, participants were placed in the VE in front of the end of a 
corridor (facing a wall), and asked to judge the distance to the wall. There was no corresponding real world condition. 
As shown in Figure 6, the narrow field of view condition (60 degrees horizontal x 38.5 degrees vertical) produced 
worse estimates than the wide field of view condition (140 degrees horizontal x 90 degrees vertical). It was the only 
condition in these experiments in which distances were overestimated in VE. A probable cause is the that the narrow 
field of view causes the loss of perspective cues which are important for distance estimation in this situation. 



Figure 6. Distance estimates as a function of viewing 
distance and FOV 


VE DISPLAYS AND PERFORMANCE ON OTHER TASKS 

Other experiments have looked at how display characteristics affect performance on tasks other than distance 
estimation. Lampton, Gildea, McDonald, and Kolasinski (1996) compared a monitor, the BOOM2C, and the Virtual 
Research Flight Helmet in terms of performance on five tasks. Body movement and orientation were controlled by a 
joystick, with head coupling available for the HMD and Boom conditions only. The tasks were: 

• Search - Searching a room for a flying object while seated in the center of the room. 

• Turns -- Moving through a narrow corridor which made a series of alternating left and right turns. 

• Doorways —Moving through a series of rooms. 

• Tracking —Moving a joystick-controlled pointer to a fixed target. 

• Bins -Using a joystick to place an object in the correct bin. 



VEPAB Task 


Figure 7. Time to complete selected VEPAB 
tasks as a function of display device 
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As shown in Figure 7, there were significant differences among these devices on only one task, the search task. 
Today’s college students, perhaps because of their experience with video and computer games, appear to be quite 
proficient in searching a room while viewing it in a monitor and controlling their point of view with a joystick. The 
data are also a reminder that adding somewhat unstable mass to the head tends to affect head movements. 

Simulating Locomotion in VE 

Lampton, Knerr, Goldberg, Bliss, Moshell, and Blau (1994) compared the joystick and a spaceball to control 
locomotion through a variety of simple environments. Some involved “flying”, and others did not. On every 
locomotion task, movement was significantly faster with the joystick (Figure 8) 



VEPAB Task 


Figure 8. Locomotion task performance 
as a function of control device 



Figure 9. Error in distance estimates as a function 
of locomotion method and compensatory cueing 


They attempted to keep travel time constant for all groups. The joystick moved participants forward at a fixed speed 
when pushed forward. Teleportees waited the same amount of time as moving the route by joystick required before 
they were teleported. Treadmill participants were instructed to maintain a fixed speed, and were automatically given 
recorded verbal reminders if their movement varied from that standard. When the auditory cues were present, the three 
groups did equally well. In the absence of auditory cues, the joystick group did best, and the treadmill group did the 
worst. The visual movement cues that the joystick group received are believed to be responsible for their superiority to 
the teleport group. The poor performance of the treadmill group is attributed to the workload required by the secondary 
task that they were given, that of maintaining a constant speed. 

While much of the research has taken place in the simulated indoors, both simulated and real outdoor terrain were used 
in the investigation of spatial knowledge acquisition in VE. In the first experiment (Singer, Allen, McDonald, and 
Gildea, 1997), college students navigated one of two terrain areas, stopping at various locations along the route to 
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learn to identify various landmarks. After the training, they were tested by being placed at new locations and asked to 
point to and estimate the distance to the same landmarks. There were three training conditions: 

• High VE, using a head coupled HMD and the treadmill 

• Low VE, using the same HMD, but without head coupling, and a joystick 

• Map study 

The performance of the college students, in terms of landmarks correctly located, was straightforward. The Hi-VE 
condition was significantly better than the map condition, with the Low-VE condition somewhere in between 
(Figure 10). 



Training Mode 


Figure 10. Percentage of landmarks located 
correctly as a function of training mode 


In the second experiment (Singer, Allen, McDonald, and Fober, in preparation), soldiers at Fort Benning, GA 
performed the same task using a moderate fidelity representation of an actual training area. They used either map study 
or a mid-level VE interface (head-coupled HMD with joystick control of movement). In addition to being tested in the 
VE, they were also tested on the actual terrain. The performance of soldiers (Figure 11), when tested in the VE, is 
consistent with that of the students, although there was only one VE condition. The VE group did better than the map 
group. 



VE Test Transfer Test 

Figure 11. Number of landmarks correct identified 
by soldiers in VE and on the actual terrain 

However, on the transfer test, which was performed on the actual terrain, there was no difference between the groups. 
Moreover, as is common in studies of transfer of training from simulation devices, both groups did worse on the 
transfer test than they did on the VE test. There are several possible explanations for this. First, the soldiers were all 
Army officers who were experienced at relating maps to terrain, but not at relating computer-generated visuals to 
actual terrain. Second, perhaps the terrain database was not sufficiently like the actual terrain. While it was 
topographically correct, the color and visual texture were not necessarily correct. Generic plants were photo-textured, 
and were not the correct height or color. Some re-grading and planting was done, and vegetation grew, between the 
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time the information for the database was collected and the experiment was conducted. Participants may have 
incorrectly focused on the specific vegetation cues, rather than the terrain contours and roads. This suggests that either 
lower fidelity (obviously generic vegetation) or higher fidelity (more like the actual views) approaches might be more 
effective. In contrast to exterior terrain, Witmer, Bailey, and Knerr (1996) found very good transfer of knowledge 
about building interiors from VE to the real world. 

CONCLUSIONS 

Conclusions Regarding Visual Display Systems 

• VE performance is worse than real world performance on a variety of visual tasks. Contributing factors are likely 
to be FOV and resolution. The consequences of limited FOV are less well understood than those of limited 
resolution. 

• Distances in VE are typically underestimated. 

• Stereoscopic displays improve performance only at short distances. 

• Head coupling may or may not improve performance, depending on the task performed. 

• Image fidelity is a consideration for transfer. 

Conclusions Regarding The Simulation Of Locomotion 

• The method used to control self-motion in VE can affect the perception of distance, but not always in the ways we 
would have predicted. 

• Compensatory cues can improve distance perception in VE. 

• Because of the influence of computer and video games, young people may be generally proficient in the use of a 
joystick, and may find it a very “natural” means of locomotion. 
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INTRODUCTION 

The goals of the Virtual Environment Training Technology (VETT) program are to develop, demonstrate and evaluate 
the use of virtual environment (VE) component technologies for navy training applications. A key part of this effort is 
the development of a testbed, performed by Management Systems and Training Technologies (MSTT), for investigating 
the benefits of using VE interfaces to improve naval training while reducing training-system costs. This paper describes 
the technology components integrated by MSTT that provide a tool for training effectiveness research at the Naval Air 
Warfare Center Training Systems Division (NAWCTSD) in Orlando, Florida. The development approach used for the 
testbed is to integrate off-the-shelf tools into one cohesive software platform. Applications developed on the testbed 
include a multi-modal Virtual Electronic Systems Trainer, a spatialized audio location task, and relative motion 
experiment for surface ship handling. Current efforts on the testbed are directed towards open-water and restricted 
maneuvering surface ship handling. 

BACKGROUND 

Many VE-based testbed systems have been developed and reported on in the virtual environment research community 
(Appino, Lewis, Koved, Ling, & Codella, 1992, Pausch, Burnette, Conway, DeLine, & Gossweiler, 1994, Bricken, & 
Coco, 1993, Shaw, C., Green, M., Liang, J., & Sun, Y. 1993, Davidson, 1996, Davidson, 1997). A review of such efforts 
provides a baseline in developing requirements for a rapid-prototyping, reconfigurable software testbed. Based on past 
research. Table 1 reveals some key features that should be considered in the development of a VE-research testbed. 


Performance 

Design 

SS Smoothly generated images 
-Database organization 
-Efficient processing by renderer 

Performance Measures 

-Data collection, recording, playback and analysis 
of session data 

Lo Low tracking latency 

-Communication overhead minimized 
-Predictive techniques implemented 
-Continuous sampling makes tracking data readily 
available 

Co Reconfigurable 

-Database-driven models/process configuration 

Low event latency 

-Sensory events are synchronized without noticeable 
delay to the user 

Device Independence 

-Database definitions of device 
types/mounting/units of measure 
-API is free of device dependencies 

Di Distributed processing 

-One machine/processor per task 

Portable representations across heterogeneous 
platforms 

-ASCII file definitions of various representations 

Ef Efficient communication 
-Low overhead 

Extendible database tools for adding new properties to 
objects 


Table 1. VE Testbed Performance and Design Criteria 
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VETT TESTBED FUNCTIONAL COMPONENTS 


MSTTs approach in creating the VETT software testbed is to use software components from Division Inc., Entropic 
Research, SensAble Technologies, Paradigm Simulation, and Dynasim AB to provide the needed software capability. 
Division Inc. provides the necessary database infrastructure, communications, and processes to create basic VE 
applications. Entropic Research provides tools for speech recognition. SensAble Technologies provides hardware and 
software to simulate sensory input involving touch. Dynasim AB specializes in object-oriented simulation and provides 
the needed capability to simulate symbolic systems of equations, and Paradigm Simulation provides DIS and marine 
effects capabilities. C++ is used for in-house process development, and UNIX Systems Lab’s C++ Standard 
Components Library is used to provide common tools for software development. 

The most difficult aspect of testbed development is the integration of other subsystems so that the use of added features is 
transparent to the user and software developer. Figure 1 shows all the major functional components of the VETT 
testbed. The simulation system, speech recognizer, haptic interface, data collection, record/replay, and augmentations to 
the visual Tenderer are all new servers that have been added to the basic VETT kernel. The following sections provide 
brief descriptions of the testbed functional components. 



' } 

\ x 

under development 


IRIX Operating System/Windows NT 


USL C++ 

Standard Components 


Figure 1. VETT Testbed Software Components 


TESTBED INFRASTRUCTURE (Division Inc.) 

dVS Runtime . The VETT testbed infrastructure is built on the dVS Runtime/dVISE software environment from Division 
Inc. It provides a standard suite of “Actors” that are typically required by all VE applications. The dVS Runtime 
includes processes to support visual rendering, sound spatialization, collision detection, 3D position tracking, and the 
virtual body. Each Actor is reconfigurable, and the interface to control the various processes of the dVS Runtime is 
common across a number of different platforms (Division, 1996). 

The dVS Runtime environment provides facilities to write user-developed applications as well as the ability to create 
third-party Actors which can be integrated with the dVS Runtime. Additionally, Division provides a shared-database 
communication scheme which is available to third-party Actors. The interprocess communications (IPC) is based on 
semaphores and shared memory of the native operating system. Consequently, Division’s IPC is high-speed and can be 
used for periodic or asynchronous updates. Division also supports networked, collaborative VEs. 

dVISE. A Virtual Data Interchange (VDI) database is used as input to Division’s interactive VE authoring tool, dVISE. 
The VDI script is an ASCII database which is used to describe objects in the VE as well as events that occur as a result of 
user or object interactions. Objects can be inherited from VDI libraries and augmented with additional behaviors. The 
authoring/user environment supports many different 2D and 3D peripheral devices and allows the participant to create 
VEs interactively. Division also provides the ability to expand dVISE with user-defined events which makes it possible 
to tailor dVISE to meet the user’s unique requirements. 
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FORCE-FEEDBACK SYSTEM (SensAbie Technologies) 

The haptics capability of the VETT testbed comes from SensAbie Technologies’ PHANToM haptic interface and Ghost 
software development kit (SDK). The Haptics Actor is designed to fit in with the other Actors that make up the dVS 
Runtime. The design assumes that the definition of the haptic database is external; haptics attributes, defined in the VDI 
database, are attached to entities in the VE. The Haptic Actors must monitor these entities and reflect their state (position 
etc.) within the haptic database. The Haptics Actor also monitors and reports the position of the PHANToM input stylus 
to the dVS Runtime environment (Division, 1997). 

SIMULATION SYSTEM (Dynasim AB) 

Dymola . Dymola, from Dynasim AB, is an object-oriented software tool for modeling large continuous-time systems 
and discrete events. The tool allows the modeler to create simple “base” classes that abstract shared physical interfaces 
between various components in a large system. The base classes are typically used to define what a “connection” is 
between two or more components (Elmqvist, 1994). 

Dymosim . Dymosim is C program text that is generated from the Dymola language description. Additionally, Dynasim 
AB provides utilities to create a real-time version of Dymosim that contains fixed-step size integration routines, a shared- 
memory API for control, and access to all model parameters, inputs, outputs, and state variables via shared-memory. 
Configuration files are used to select the integration scheme, initialize variables, and determine the duration of the 
simulation for non real-time applications. Additionally, any number of processes can be attached to shared memory. 
This is useful for displaying the results of the simulation to the participant in the VE or for data collection. 

VISUAL SYSTEM (Paradigm Simulation Inc.) 

Vega . In order to provide a realistic environment to conduct ship handling experiments, it was necessary to add marine 
effects to the testbed capability. Instead of designing a new visual Actor from scratch, it was decided to augment 
Division’s existing visual Actor with the marine special effects of Vega Marine provided by Paradigm Simulation. Vega 
application definition files (ADF) are used to configure the entire Vega system by incrementally loading ADFs based on 
environment variables. Once the Vega system is configured, all system definitions are resolved to ensure that the loaded 
configuration is valid. 

VR-Link . Although DIS is not required in current testbed applications, a DIS interface is now possible via the DIS 
option of Vega. Vega DIS is built on top of Mak Technologies VR-Link. Details as to the full implementation are still 
being considered. 

SPEECH RECOGNITION (Entropic Research) 

HAPI/HTK . Entropic Research’s HAPI speech recognition system was added to the testbed as a needed capability to 
command various controls of an application using voice. HAPI provides all the features of a modem speech recognizer 
including acoustic speech models based on Hidden Markov Models, pronunciation dictionary, language models, and 
GUI-based tools for constructing application grammars (Odell,, Ollason, Valtchev, & Whitehouse, 1997) 

The Speech Recognition Actor is classed as an Input Actor, taking as input vocal utterances and converting them into 
sequences of key-press events that are reported using the conventional dVS input stream. This input stream is monitored 
by other Actors in the dVS Runtime, including dVISE. Currently, only tagged ASCII characters are used as input to 
dVISE from the Speech Actor. Future plans for natural language processing are being considered. 

DATA COLLECTION/RECORD/REPLAY 

Data Collection . The Data Collection Actor is designed as a configurable Actor with interfaces to the dVS mntime 
environment and the Dymosim shared-memory arena. Data collection is broken out into information that is periodic and 
aperiodic. Periodic data comes from two different sources: 1) the simulation, and 2) the participant’s body movements. 
Aperiodic data is generated primarily from the participant’s interactions in the VE via dVISE. 
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Record/Replay. As a means of data elicitation and subject feedback, a Record/Replay Actor is currently under 
development. The ability to record a session with playback is complete. Future plans include first or third person 
review, fast forward/reverse, pause, and the ability to step into the simulation during replay. 
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ASSESSMENT AND EXPERIMENTAL APPLICATIONS OF VIRTUAL 

REALITY TECHNOLOGY 

Day Chairman: Peter Werkhoven 

Head of Dept, of Information Processing 
TNO Human Factors 
P O BOX 23 

3769 ZG Soesterberg, The Netherlands 

NOTES ON DAY 3 OF THE RSG28 WORKSHOP 


INTRODUCTION 

The subject of Day 3 of the December 1997 Workshop of the RSG 28 was Assessment Methods and Military 
Applications. This subject followed logically on the subjects discussed during the previous two days of the workshop 
(military requirements on Day 1 and human factors issues on Day 2). 

During the introduction of Day 3 the chairman outlined 

• That a thorough task or training analysis is required to identify the potential use of VR by the military, 

• That a comparison of alternatives (desktop/simulators/VR) was required to get optimal solutions with respect 
to cost/benefits, 

• That a thorough evaluation was required for quantifying the benefits and success of applications. 

Day 3 included these issues. In particular, the speakers talked about which military requirements were met, how VR was 
applied, which human factors were involved, what evaluation results they obtained and what future plans they had. Also, 
assessment methods were presented: what was being assessed and how, strength and weaknesses and potential uses. 

The program also included telepresence applications (e.g., remote operations) which show interesting similarities with 
VR except that the environment is not computer simulated but camera generated. 

PROGRAM 

Mr. P. Dumanior (US) : 

Dismounted Warrior Network 
Dr. D. Ryan-Jones (US): 

Explosive Ordnance Disposal 
Maj. B. Taylor (UK): 

Remote Controlled Vehicle “Wheelbarrow” 

Dr. J. Leddo (US): 

An Intelligent Tutoring Simulation For Military 
Operations In Urban Terrain 
Dr. N. Cislo (FR): 

Telepresence And Intervention Robotics 
Dr. L. Magee (CAN): 

Exploratory Use Of VR Technologies For Training 
Helicopter Deck-Landing Skills 
Dr. B. Loftin (US): 

Shared VE For Collective Training 
Dr. P. Hue (FR): 

French Military Applications Of VR 
Dr. R. Hays (US): 

Formative Evaluations Of The VESUB Technology 
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Mr. F. Kuijper (NL): 

HMD-Based VE For Military Training - Two Cases 
Some general conclusions can be drawn from the presentations listed below. 

REASONS FOR USING VE 

Some important reasons for using HMD-based virtual environments and not other simulation techniques (e.g. large 
screens displays) are 

• The need for 360 degrees around viewing, realism and time pressure (Dumanior) and 

• The need to continuously change viewpoints (walk around objects), or to do 3D manipulations (Ryan Jones) 

SHOWN BENEFITS OF VE 

• Working with equipment can be trained in time before delivery of the equipment (Taylor) 

• Significant reduction of equipment cost (Taylor) 

• More efficient training because in VE a higher density of relevant events can be simulated than can occur in 
the real world. Furthermore, in VE training is more flexible and can be focused on specific training needs 
(Magee) 

• Collective distributed team training can be realized in a cheaper and more effective way via shared VE 
(Loftin) than in real environments (Loftin) 

PROBLEMS ENCOUNTERED 

General problems mentioned most were: 

• Accuracy, calibration and stability of tracking devices (Dumanior) 

• A too low resolution and too small field of view in HMD applications (Dumanior) 

• The fact that verbal and non-verbal communication between users of a shared VE is poorly 
facilitated (Loftin) 

• Conservatism of potential users (Ryan-Jones) 

• Security issues, for example when classified manuals have to be digitized and integrated in the VE 
(Ryan-Jones) 

• The long time it takes to model VE (Ryan-Jones) 

• Lack of intelligent “avatar behavior” in VE (Magee) 

• Lack of configurable scenario generators (Taylor) 

• Lack of natural intuitiveness for using hands-free locomotion interfaces 

ISSUES NOT ADDRESSED ON DAY 3 

The spectrum of applications presented was limited to Army and Navy applications and did not include Air Force 
applications such a VE or augmented reality for supporting pilots during complex navigation tasks. 

The applications discussed mainly focused on training, whereas VE has also much potential as a new interface for 
operating tasks such as damage control and command and control. 
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DISMOUNTED WARRIOR NETWORK EXPERIMENTS 

Traci A. Jones 

Simulation, Training and Instrumentation Command (STRICOM) 

12350 Research Parkway 
Orlando, FL 32816, USA 

Phone (Commercial): + 1 407-384-3927 - Fax: + 1 407-384-3830 
Email: jonest@stricom.army.mil 


INTRODUCTION 

The Dismounted Warrior Network (DWN) project began in November 1995 with the publication of the STRICOM 
Individual Combatant Simulation Technology Transfer Plan [1]. Sponsored by the US Army Infantry Center (USAIC) 
and the US Marine Corps (USMC), the purpose of the first phase of DWN was to provide a means of understanding 
dismounted infantry simulation requirements. A series of experiments and exercises were planned, to include an 
assessment of the cost and benefits associated with the technologies to immerse the individual combatant within the 
virtual battlefield. One of the most important products of DWN is a Simulation Task Analysis, which was developed to 
support simulation for IC in the Training, Exercises, and Military Operations (TEMO), Advanced Concepts and 
Requirements (ACR), and Research, Development and Acquisition (RDA) domains. A preliminary technology analysis 
was conducted to assess and select current representative IC simulation technologies for inclusion in the DWN 
experiments and exercises. 

APPROACH 

DWN is a STRICOM-led project executed under an Advanced Distributed Simulation Technology (ADST) II delivery 
order, with Lockheed Martin Corporation as the prime contractor. In January 1996, STRICOM assembled a team of 
technical experts from Government and industry to evaluate the current state of the art in IC simulation. The team visited 
several technology providers and assessed the maturity of the simulations as they supported the DWN needs. 

As the focus for the DWN experiments was on virtual simulation, the systems were required to be real-time, man-in-the- 
loop, and DIS compatible so that several platforms could be networked. Among the system components reviewed and 
assessed were: body movement tracking, visual animation of human figures, directional sound, surrogate weapon 
systems, locomotion simulation, human gesture recognition, natural language processing, SAF capability and analytic 
modeling. Several contractors proposed innovative solutions for inserting the individual into the synthetic environment. 
However, based on budget and time constraints, STRICOM preferred that the systems be developed and actively funded. 
The technologies selected were grouped into four Virtual Individual Combatant Simulators (VIC), each capable of 
providing a single individual the ability to interact within the virtual battlefield with other VIC and platforms in a DIS 
exercise. 

In addition to the VICs, a Dismounted Infantry (DI) Semi-Autonomous Forces (SAF) capability was required to support 
more than a fireteam level battle (i.e. to round out the squad/platoon with computer generated forces). The DWN team 
investigated a number of different SAF systems that model DI. Among the technical capabilities reviewed during the 
SAF evaluations were: the SAF development language, platform on which the SAF runs, support of DIS protocols, 
human figure animation, ModSAF basis, voice recognition, gesture recognition, and interaction with 3D terrain. 

Following the technology assessments and selection, the DWN delivery order was awarded in June 1996. Several 
Technical Interchange Meetings were held among the Integrated Product Team (IPT) to design the experiments and 
exercises. Following a brief integration period, the engineering experiments were conducted during a three-week period 
in April-May 1997 at the Orlando, FL Lockheed Martin facility. The VICs were relocated and the user exercises were 
conducted during a three-week period in May-June 1997 at the Land Warrior Test Bed, Ft. Benning, GA. 

OBJECTIVES 

Simulation Task Analysis. Among the key objectives of the DWN project is to develop a Requirements Document for 
US Army and USMC IC simulation. The DWN experiments and exercises were conducted to provide a proof of concept 
for the technologies that enable the individual to interact within the virtual battlefield. The DWN Simulation Task 


Paper presented at the RTO HFM Workshop on "The Capability of Virtual Reality to Meet Military 
Requirements ”, held in Orlando , USA , 5-9 December 1997[ and published in RTO MP-54. 
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Analysis was developed to assist the Army and USMC soldiers in defining the requirements for IC simulation in the 
ACR, RDA and TEMO domains. A Functional Definition Process was developed, based on the Army’s Close Combat 
Tactical Trainer (CCTT) Program, with changes necessary for the ACR and RDA domains. Linkage was provided to 
existing Army and USMC doctrine and tactics. Individual to squad, squad to platoon, and platoon to company 
organization level tasks were analyzed for the domains. 

An extensive series of documents were systematically developed as part of the Simulation Task Analysis and became the 
basis for the DWN Requirements Document. This documentation provides traceablility back to doctrine. A relational 
database was developed to allow direct Validation, Verification and Accreditation functionality. The documents include 
the following: Functional Definition Process, ACR/RDA Functional Definition, TEMO Functional Definition, 
ACR/RDA Functional Fidelity, TEMO Functional Fidelity and Requirements Document. The information can be viewed 
and downloaded by accessing the DWN home page at http://www.rciorl.com./htms/warrior.htm. The intent of the 
Requirements Document is to form the basis for follow-on virtual IC simulations and simulators required by the Army 
and USMC. 

DoD Defense Technology Objective (DTO). DWN directly supports DTO IS.40.01, Individual Combatant and Small 
Unit Operations (IC/SUO) Simulations in the Synthetic Environment. This joint DTO is led by the STRICOM 
Engineering Directorate. Among the objectives is to develop and demonstrate technologies for creating real-time 
simulations to immerse the individual soldier and allow for interaction in the synthetic environment. The products that 
will evolve within the DTO include simulations which are High Level Architecture (HLA) compliant and that support 
low cost solutions for mission rehearsal, materiel development and training of individual soldiers and marines. There is 
also potential application of these technologies to training and rehearsal for the FBI and law enforcement communities. 
DWN and its results will form the basis of the virtual simulation capabilities for the IC/SUO DTO. 

EXPERIMENTS AND EXERCISES 

Technologies 

Four VICs and DI SAF were integrated and evaluated on two digital terrain databases during the DWN engineering 
experiments and user exercises. As mentioned above, these technologies were chosen based on maturity, capability, 
availability and their contribution to form a representative and complementary sample of existing technical approaches. 
Figure 1 provides a System Block Diagram for DWN. 



Figure 1. DWN Block Diagram 


As shown, the system consisted of four VICs, a BLUFOR station comprised of DI SAF, Exercise Support equipment, 
and an After Action Review (AAR) capability. All systems were connected via an ethemet network and communicated 
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with each other via DIS 2.0.4 PDUs. A wireless radio was provided for communications between the soldiers on the 
operating VICs. 

The four VICs provided a wide variety of techniques for accomplishing the basic functions of locomotion, display, and 
weapon aiming as shown in Table 1. 


Function 

VIC Alpha 

VIC Bravo 

VIC Charlie 

VIC Foxtrot 

Locomotion 

Human Joystick 

ODT 

Joystick 

Foot Pedal + 
Head 

Orientation 

Visual Display 

Wireless HMD 

4 Projection 
Screens (WISE) 

Desktop 

Monitor 

Single 

Projection 

Screen 

Body Motion 
Capture 

Video 

Magnetic 

N/A 

Magnetic 

Weapon Tracking 

Video 

Magnetic 

N/A 

Acoustic 

Weapon Aiming 

Video thru HMD 

Video thru IHAS 

Video 

crosshairs on 

CRT 

Rifle sight on 
rear projection 
screen 

Directional Sound 

SOUNDSTORM 

3d 

SOUNDSTORM 

3d 

Yes 

Yes 

Human Animation 

Biomechanics 

DI-Guy 

JackML 

DI-Guy 


Table L VICs Comparison Matrix 

VIC Alpha is the Dismounted Soldier Simulation (DSS) system developed by Veda, Inc. under a STRICOM Broad 
Agency Announcement contract [2], [3], [4]. Unique technologies utilized by this system are the optical (camera) based 
full-body motion capture system, the use of a wireless head-mounted display (HMD) and the ‘human joystick’ paradigm 
for locomotion. VIC Alpha is the only completely untethered DWN system that allows the soldier to freely maneuver in 
a pre-defined space. 

VIC Bravo is a combined system consisting of Virtual Space Devices, Inc. (VSD) Omni-Directional Treadmill (ODT) 
mobility platform combined with Reality By Design’s (RBD) Bayonet visual simulation system. Bayonet is based on 
NPSNET-IV from the Naval Postgraduate School with additional RBD enhancements, extensions and optimizations 
specifically for IC operations [5], [6], [7]. Unique technologies presented by this system include an omni-directional, 
force-feedback mobility platform allowing full 360-degree operation. The Bayonet system presents a 270 degree field- 
of-view visual display using a Walk In Synthetic Environment (WISE), combined with a monocular monochrome HMD 
simulating the video camera from the Land Warrior (LW) system. Bayonet utilizes magnetic based tracking for weapon 
aiming. 

VIC Charlie is the TRAC-WSMR Soldier Station [8], [9]. Soldier Station is a combined virtual/constructive simulation 
solution focused on analytic modeling and evaluation. Unique technologies include the use of a joystick and touchscreen 
user interface, with a computer monitor display. Soldier Station uses validated constructive algorithms from Janus 
combined with an enhanced version of NPSNET-IV for visuals, networking and the user interface. The fully functional 
Soldier Station was used during the user exercises only. 

VIC Foxtrot is the USMC Team Tactical Engagement Simulator (TTES) developed at the Naval Air Warfare Center 
(NAWC-TSD). TTES presents a high-resolution virtual image via a large rear-projection screen. Locomotion velocity is 
controlled using a pressure sensitive foot pedal. The user’s head is tracked with a magnetic sensor and is used to control 
steering through the environment. A de-milled rifle is used for aiming and firing and is tracked using an acoustic sensor. 

Based on the data collected and analyzed during the technology assessment, a Dl-oriented SAF system did not exist with 
Army behaviors suitable for the DWN experiments. An Army DI SAF capability was therefore developed by SAIC 
based on the USMC IC SAF, which had supported Leathemet as part of the DARPA STOW program [10]. The USMC 
IC SAF behaviors were modified to reflect standard Army doctrine and an additional twelve behaviors were added to the 
original set of maneuvers. The DI SAF will be integrated into ModSAF 4.0. 
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Two digital terrain databases were selected based on the high fidelity requirements for IC operations. The Range 400 
database of 29 Palms, CA was selected based on its dense resolution and open terrain features. The McKenna Military 
Operations in Urban Terrain (MOUT) database of Ft. Benning, GA was also selected based on its high resolution and 
representation of a built-up area. Both databases are Government furnished managed by the US Army Topographic 
Engineering Center (TEC). All VICs used a visual representation of the databases in the industry standard Multigen 
Flight format. DI SAF used a correlated version 5 Compact Terrain Database (CTDB) format database ot the same 
regions. 

ENGINEERING LEVEL EXPERIMENTS 

The engineering experiments were conducted at the Lockheed Martin Information Systems facility in Orlando, Florida 
from 19 ApriMO May 1997. The purpose of the experiments was to assess the capabilities of current IC simulation 
technologies in support of future training, virtual prototyping and evaluation of new weapons systems. 

Eight soFdiers from Ft. Benning, GA served as subjects for the experiments. Each soldier utilized and experienced all 
VICs and scenarios. Three days of training and testing per soldier were conducted per VIC. The U.S. Army Research 
Institute (ARI) collected data and provided an independent evaluation of the experiments. Data was collected by the 
following means: DIS PDU data logging, Pre- and Post-exercise subject debrief and rating assessments, and videotaping. 
Measures of performance were specified for each experiment and were augmented by subjective ratings and 
questionnaires. Data from the experiments was analyzed using Excel and statistical analysis software [11]. 

A number of IC simulation issues were examined during the experiments. Table 2 provides a summary of the functions, 
issues, performance measures and tasks for the specific variables tested and analyzed. 


Function 

Issues 

Performance Measures 

Tasks 

Locomotion 

• Time to acquire proficiency 

• Level of proficiency 

• Errors over time 
(collisions) 

• Task completion times 

• Movement rates 

• Move through building; 
maneuver around obstacles 

Visual Display 

• Lag 

• Resolution 

• Object detection/identification 

• Facilitation of locomotion 
(Distance estimation) 

• Time to detect/identify 
object 

• Range at object 
detection/i.d. 

• Location of object relative 
to self/other object 
locations 

• Engineering 
measurements 

• Locate and identify 
vehicles, DI (friendly and 
foe) 

Body Motion Capture 

• Accuracy 

• Lag 

• Reliability 

• Engineering 
measurements 

• No man-in-the-loop 
component 

Weapon 

Tracking/Aiming 

• Sighting Ease 

• Tracking Accuracy 

• Lag 

• Accuracy of shooting at 
static and moving targets 

• Time to engage 

• Engineering 
measurements 

* Shoot at static and moving 
targets 

Directional Sound 

• Object localization 

• Speed and accuracy of 
assessment of object 
direction and location 

• Locating noisy objects in 
environment 

Human Animation 

• Lag 

• Realism 

• Engineering 
measurements 

• Subjective assessments 

• Observe DI figures 
walking, running, crawling, 
etc. 

Miscellaneous 

General Issues: 

• Comfort, Simulator sickness 

• Noise, Safety 

• Immersion 

• Engineering 
measurements 

• Subjective assessments 

• Questionnaires 

• Observe during execution 
of all tasks 


Table 2. Engineering Experiment Design 
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The intent of the Engineering Experiments was to compare the various component technologies represented in the four 
VICs over different tasks. The results were documented in order to be matched against functional fidelity requirements 
flowing from the simulation task analysis portion of DWN. The results will provide the beginnings of a database that 
will match existing technologies and capabilities against simulation requirements, and the identification of areas where 
future technology development is required. 

User Level Exercises 

The user exercises were conducted from 27 May-13 June 1997 at the USAIC Dismounted Battlespace Battle Lab 
(DBBL) using the Land Warrior Test Bed (LWTB) facilities, Ft. Benning, Georgia. The purpose of the user exercises 
was to focus on the system level capabilities of the technologies, whereas the engineering experiments focused on the 
technology components. The ability of the soldiers to operate on the VICs to support collective and individual 
dismounted infantry tasks was evaluated. 

Two basic scenarios were conducted — an open terrain attack/defend operation on the Range 400 database and a building 
clearing operation on the McKenna MOUT database. All operations involved a platoon of friendly forces consisting of 
the VICs operating as a four-man fireteam, together with a second fireteam of SAF. All friendly entities were 
commanded by a platoon leader using a joystick-driven version of Bayonet. All operations were simulated under 
daylight conditions. During each exercise, one of the VICs would serve as the fireteam leader. 

The open terrain scenarios involved two basic variations — an attack on an opposing force (OPFOR) site and the defense 
of a friendly site against an advancing OPFOR. Minimal cultural features are available on the Range 400 desert database 
so the soldiers were forced to make the most use of the terrain features for cover and concealment. Standard Army 
doctrine was used to conduct the operations including the use of bounding overwatch and suppressive fire. The end of an 
exercise was declared to occur when either all enemy or friendly forces were eliminated. 

The MOUT scenarios involved the friendly forces clearing opposing forces from a village. Operations were conducted 
with enemy placements both inside and outside of the structures. Several scenarios involved an enemy sniper located in 
multi-story buildings thus requiring the friendly soldiers to enter the structures, advance down narrow hallways and 
proceed up and down stairways. The sniper was played by a man-in-the-loop using a Bayonet station. The end of an 
exercise would be declared when all enemy had been eliminated as a threat or the friendly forces were unable to continue 
due to casualties. 

An After Action Review (AAR) was conducted at the conclusion of each exercise, including data logged playback of the 
scenario. ARI collected anecdotal data during the exercises and subjective data during the AARs. 

RESULTS OF DATA ANALYSIS 

PDU Data Analysis . DIS PDUs, specifically Entity State, Fire, Detonation, and Collision PDUs, collected at the exercise 
support station served as source data for the majority of the analyses conducted after the engineering experiments. The 
experimental tasks were intended to investigate VIC performance in the areas of locomotion, visual system performance, 
and weapon aiming/shooting accuracy (see Table 2). The results of these tasks are described below. 

Locomotion . All VICs were set with identical maximum sustained rates of locomotion. Based on course completion 
times, Bravo was the slowest VIC because the soldier had to expend real energy to move through the environment. 
Foxtrot was the fastest of the VICs, but at a cost of significantly more collisions with building structures. Bravo, along 
with Alpha, had the fewest number of collisions. 

Visual System . VICs Charlie and Foxtrot had the highest resolution visual systems, followed by Bravo, with Alpha’s 
HMD providing the lowest resolution of all the VICs. Performance in the target search, recognition, identification, and 
animation detection tasks generally reflected these differences in resolution. In decreasing order of performance, the 
VICs ranking for these tasks is Charlie, Foxtrot, Alpha, and Bravo. 
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Secondary performance measures included target range and azimuth estimation and the detection of target motion. For 
the estimation tasks, VIC performance rankings were reversed from the primary performance measures (Bravo, Alpha, 
Foxtrot, Charlie from best to worst). A more spatially constant representation of azimuth heading in Bravo and Alpha is 
proposed to explain the performance differences in this measure; better range estimation performance is believed to be an 
artifact of fewer long-range detections. Finally, VIC Alpha and, to a lesser degree, Foxtrot performed worst at target 
motion detection. It is hypothesized that these head-coupled systems made it more difficult to isolate target motion from 
self motion. 

Weapon Employment. None of the weapon tracking/aiming technology implementations provided an adequate weapon 
aiming solution. VICs with more natural aiming methods (i.e., Bravo and Foxtrot) were better at target acquisition, but 
Foxtrot’s acoustic sensors proved the least accurate weapon tracking system. 

Subjective Data Analysis. ARI researchers collected a variety of subjective data through the administration of 
questionnaires, by observation, and by soldier interviews. Some of the primary findings include the following: 

• Simulator sickness was not a problem for the soldiers using the VICs. 

• Task difficulty ratings generally followed the pattern of results found in the PDU data: VICs that performed better 
were judged easier to use. 

• Bravo was ranked as the overall best VIC in supporting tactical realism by soldiers during the user exercises; Foxtrot 
was generally second in the various ratings 

• Coordinating movement among the VICs was considered difficult 

• Alpha’s ratings were negatively affected by a lack of system reliability 

• Charlie had some good features but was generally considered the most unrealistic 

Other issues concerning database correlation between the SAF and VICs (and between VIC Charlie and the other VICs), 
VIC virtual figure identification, extraordinary SAF lethality, and communications difficulties were noted during the user 
exercises. 

Discussion. The experimental results point out the relative strengths and weaknesses of the VICS and DI SAF. Weapon 
tracking is still problematic - none of the three technologies used provided the required accuracy. The case was made 
once again for better display resolution and field of view. Locomotion technologies need to provide for rapid transverse 
of open terrain as well as precise maneuvering inside of buildings. VICs need to appear natural, easy to use, and reliable 
to gain soldier acceptance. These lessons learned provide the evolutionary forces to shape future VICs and SAF 
development efforts. 

FUTURE EFFORTS 

Follow On Enhancements 

Based on the preliminary technology assessment and observations made during the experiments and exercises, several 
components or technologies are missing that are needed for realistic dismounted infantry operations. Some clear 
deficiencies have been identified and others have been revealed by the analysis of the experimental data. 

None of the VICs utilized in DWN have the ability for the soldier to command and control SAF directly using voice or 
gestures. All SAF operations must be controlled with an additional man-in-the-loop 'battle master’ to issue instructions 
and orders to the SAF entities. An intuitive and direct interface to allow command of SAF by a real soldier-in-the-loop is 
desired to enhance the reality of scenarios. 

Communication between the soldiers on the VICs was accommodated by a wireless radio technology. While this 
approach did allow the soldiers to talk, it simulated a basic radio capability that individual soldiers do not have today in 
the field. In addition, the soldiers were not able to sense distance or location to the originator of the transmission as they 
would in the real world. As soldiers become more equipped in the future, the capability is needed to more accurately 
simulate communications between individual soldiers and link the soldiers with higher command, including digital 
messaging. 
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As noted, individual combatants need to perform several activities that are not currently fully supported by any of the 
current VICs. Some of these activities include: throwing a grenade, setting and detonating explosive charges to gain 
entrance to a building, scaling walls and rappelling down ropes. These tasks and others are identified by the simulation 
task analysis and warrant further research and development. STRICOM, together with other Government and industry 
partners, is continuing to identify and work toward solutions for individual combatant RDA, ACR and TEMO simulation 
requirements for the future. 

SUMMARY 

The STRICOM Dismounted Warrior Network program is designed to identify the requirements and capabilities related 
to man-in-the-loop, networked systems to support individual combatant simulation. The simulation task analysis phase is 
oriented toward defining a set of validated requirements and functional fidelity needed for soldiers using technology to 
support the RDA, ACR and TEMO domains. The engineering experiments and user exercises were an attempt to gather 
relevant data to determine which technologies and systems are useful today and identify areas where significant future 
work is needed. Several representative technologies were integrated to provide technical insight regarding issues 
concerning IC simulation. Based on analysis of the data collected, the US Army is focusing future efforts on the 
identified technologies that require enhancements to meet realistic IC simulation capabilities. 
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APPLICATION OF VIRTUAL ENVIRONMENT (VE) TECHNOLOGY FOR 
EXPLOSIVE ORDNANCE DISPOSAL (EOD) TRAINING 

David L. Ryan-Jones* and Eleanor Robinson 
Navy Personnel Research and Development Center 
San Diego, CA, USA 

The primary job of an EOD technician is to deal with unexploded ordnance. There are over 20,000 different technical 
publications covering items in the ordnance inventory. The large number and variety of ordnance effectively precludes 
memorizing the identifying characteristics and render-safe and disposal procedures for more than a few common items. 
Although ordnance can be blown up in place, each technician must be trained to disarm and dispose of ordnance as if the 
conditions do not allow it to be blown up in place. 

The technician’s job involves several distinct steps. First, the technician must conduct a site reconnaissance to determine the 
tactical situation. This may be performed personally or with a remotely operated vehicle (ROV). Next, the technician 
attempts to identify the unexploded ordnance from visible features. This can be very difficult if the ordnance is damaged or 
partially buried in the ground. Identification can also be difficult if the ordnance is an uncommon inventory item, or when it 
is a new model of an existing item. As a result of these factors, there may be a great deal of uncertainty about the identity 
of the ordnance. 

But, once the ordnance is tentatively identified, the real work begins. The technician must match the ordnance with the 
Series 60 technical manual that contains the render-safe and disposal procedures. The technician will then review the 
associated Series 60 publication, and memorize the render-safe procedures. Because manuals are in a traditional text with 
graphics format, it is not possible to practice EOD render-safe procedures before they are performed. This makes the job 
more dangerous than it should be. Other than technical publications, there are few aids available to the technician for 
procedural review and practice. 

Once the render-safe and disposal procedures are memorized, the technician must select the tools required to perform the 
procedures, and then actually disarm and dispose of the ordnance. The tools of the trade are quite varied, and part of 
evaluation in training is the correct selection of the tools. It can reasonably be presumed that if the technician develops a 
good mental model of the disarming and disposal procedures, the work will proceed in a safe and uneventful manner. 

Procedural skills are believed to be highly perishable. Frequent practice and review is normally required to reach and 
maintain acceptable levels of performance on complex procedural tasks. But again, the very nature of the ordnance universe 
does not support traditional training techniques. EOD tasks involve coordinated use of sensory, cognitive, psychomotor, and 
memory faculties, so one would expect that the EOD training environment would be very realistic, and the standard for 
realism is the ordnance itself. Ordnance disposal is analogous to just-in-time training. That is, a technician trains on a 
procedure just before the procedure is to actually be performed. 

In the school setting, training may be accomplished by the use of live and inert ordnance, or with mock-ups. But the use of 
live or inert ordnance may not be the best way to train for correct identification, disassembly methods, or render-safe 
procedures. It is inherently unsafe to use live ordnance for training, and realistic practice usually involves costly supervision 
and strict safety procedures. Both live and inert ordnance may be very expensive, and may be damaged or destroyed in the 
process of completing the render-safe and disposal procedures. In addition, there may not be sufficient ordnance available 
so each student will have his or her own example to practice on. The use of mock-ups can also be a problem because of 
low-fidelity and functionality problems. 

The work discussed here incorporates interactive graphical models and animated demonstrations into the technical training 
material. Prior research has suggested that realistic 3D representations are vital in learning the spatial and procedural 
relationships underlying device construction and operation, and may enhance learning of the spatial characteristics of 
ordnance, giving students the opportunity to encounter, resolve, and practice multiple options for disposing of live ordnance. 


* Currently at: Navy Air Warfare Center , Training Systems Division, 12350 Research Parkway , Orlando, FL 
32826, USA 
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VE technology seems to be well-suited for training in this context. Ordnance is clearly three-dimensional. Technicians 
must walk around ordnance, and in certain cases hold and rotate ordnance in their hands. There are great changes in scale 
from the technician’s perspective during the procedures. In addition, there are dynamic interactions between the technician 
and the ordnance that frequently involves the use of tools required to safely disarm and dispose of the ordnance. 

In spite of the promise of better training from the use of VE technology, there are problems associated with implementation 
of advanced computer technology in the military school environment. Perhaps the greatest problem is the reluctance of the 
schools to change the current system. The system is based upon traditional lecture supplemented by demonstration and 
practical exercise. The current system works well but new technology probably works just as well and requires fewer 
instructors and administrators. 

A secondary issue is the cost of creating the courseware to be implemented at the school. Much of the cost is related to the 
creation of the models used in courseware. Although engineering drawings of ordnance already exist, CAD/CAM models 
are virtually nonexistent. So, much of the cost of a virtual training systems is the expense of building the 3D interactive 
models. One technology competing with VE is animation, and embedded animation can also be used to train or review 
ordnance render-safe procedures with a much lower development cost than a fully interactive virtual model. 

In the research described here, VE technology was used in a module intended to train students to perform the render-safe 
procedure for a single ordnance item. The objective of this project was to demonstrate that EOD procedures could be 
taught by a virtual ordnance model. The specific technology used in this project was determined by the funding level, 
model development costs, command desires, and the potential for implementation. Our strategy was to select off-the-shelf 
hardware and software as much as possible to meet the stated objective of the project. 

The ordnance selected as the prototype for the project was the SUU-25 flare dispenser. This is a device by which flares that 
are intended to illuminate a battlefield are dropped from an aircraft. The dispenser may contain live flares that are in 
themselves explosive, as well as mechanisms to force them out of the dispenser. The dispenser is made safe by a sequence 
of actions that involves cutting wires, removing ejection cartridges, removing an access door, inserting a shorting pin, and 
removing any live flares. This particular piece of ordnance is the family prototype used at the EOD school. The dispenser is 
inert and attached to a helicopter in that environment. Since the device is inert, students are trained and tested by describing 
the render-safe and disposal procedures. 

Command and fiscal restraints required that training be performed on a personal computer in the form of a traditional 
computer-based training module. Authorware Pro was selected as the instructional development software, 3D Studio for the 
modeling software, and World Toolkit for Windows as the virtual environment software. At the time of initial development, 
a 133 MHz Pentium processor was state-of-the-art, and the fidelity of the model and its characteristics was reduced to 
accommodate these restrictions. The actual flare dispenser is cylindrical but the model is 12-sided to restrict the number of 
polygons to about 20,000. In addition, the model does not rotate continuously but in 10 degree steps to speed up the 
response on the screen. The slow response of the model during rotation was perceived as a problem by the development 
team but not by the users at the EOD School. As personal computer technology advances, the virtual model will become 
more responsive or the level of fidelity can be increased with the same response time. 

When the model was first tested at the EOD School the instructors suggested other changes to the level of fidelity. For 
example, screws on an access panel had to be removed one by one before the panel could be opened. The instructors felt 
this was a waste of time, and they suggested that after the removal of just one screw the door should disappear. This 
principal was applied to other aspects of the render-safe procedure. The unintended side effect was a noticeable change in 
the perceived fidelity of the model. But, the change was necessary to get the support of the instructors. 

One unique feature of the user interface is the tool kit. Every ordnance procedure requires the use of tools, and testing 
usually involves the selection of the correct tools for the procedure. The tools required to perform the procedure in our 
module are shown on the screen as icons. The user must select the correct tool to perform a step, and the cursor changes to 
the icon of the tool. If the wrong tool is selected, the step cannot be performed. During training, selection of the correct tool 
is guided by the courseware. During testing, the user must select the tool to be used at that step. Selection of the wrong tool 
or performing a step out of sequence is noted as an error. 

The virtual model is embedded in the instructional software by design, and cannot be called outside of the instructional 
module. The purpose of this restriction is to comply with the instructional philosophy of the EOD School. At the school, 
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most procedures are taught by demonstration and practical exercise, and free play is frowned upon. In fact, it is very 
dangerous in the real world to violate the stated procedures. Safety considerations permeate all aspects of instruction at the 
School. An alternate implementation suggested by the instructors is to incorporate the model into an instructor-controlled 
demonstration of procedures. 

The training module is organized around seven lessons. The virtual model is used in only two of those lessons, one for 
step-by-step training and the other for testing. The time required to complete the entire module is about forty minutes. 
However, only ten minutes are allocated at the School for training the entire family of this type of ordnance, and it 
immediately became obvious that the module would never be implemented in the present form. But, students are required 
to perform two hours of study on their own each evening, so this may be the best mechanism for formal testing. It is clear 
from our pilot test that the virtual model is as good as lecture and demonstration, but a transfer of training experiment is 
required to demonstrate that it is effective and is without negative transfer. 

In summary, we believe that virtual worlds will have an important role to play in individual training. The fact that they seem 
to work as well as traditional instructional methods should encourage us to build more applications. But practical 
consideration may severely restrict the form of the implementation, and consequently the conclusive evidence that we need 
to convince others to build more models. In addition, until the cost of building high fidelity applications drops dramatically, 
it may be difficult to find users willing to take our models and implement them in a training context. 
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REMOTE CONTROLLED VEHICLE “WHEELBARROW” 

Major Barry Taylor 
Army School of Ammunition 
Marlborough Barracks 
Temple Herdewyke 

Leamington Spa, Warwickshire CV33 OUL, U.K 


INTRODUCTION 

The Army School of Ammunition in Warwickshire, England is the United Kingdom’s premier IEDD training 
establishment. We have been training IEDD operators since about 1922 and most recently our efforts have been centred 
on the training of individuals for Northern Ireland operations. In addition we also train students from many countries 
around the world, principally in EOD matters, but also in conventional Land Service Ammunition, which is our main 
task in the Field Army. 

We are currently using and developing two computer-based training systems in the IEDD field. The two systems are: 

• RCV Wheelbarrow MK8B Control Trainer System 

• IEDD Threat Assessment Trainer 

WHEELBARROW MK8B CONTROL TRAINER SYSTEM 
Introduction 

In 1993 the United Kingdom’s prime remote controlled EOD vehicle, Wheelbarrow was subjected to a mid-life 
improvement (MLI). The aim of the MLI was to extend the in-service life of the Wheelbarrow through to the year 2000 
whilst development on its replacement took place. This upgrade programme resulted in the complete overhaul of the 
Wheelbarrow and the introduction of modem electronics technology. Whilst the general appearance of the vehicle did 
not change its functionality changed considerably. One aspect that did change was the Hand Controller . The previous 
system had been a fairly straightforward unit that was easily adapted to by operators and required very little 
familiarisation before they became proficient in its use. The replacement was, however, a much more complex system 
and with each new RCV costing in the region of £100k. In addition, there was a dedicated training requirement for the 
Hand Controller alone, before we could expose operators to the main vehicle. Of greater importance was the initial 
timeframe in which the changeover from MK8 to MK8B had to take place. Operational units therefore needed 
conversion training before they received the new vehicle. Special courses were designed to manage this turbulent period 
and, over a 6-month period, all IEDD operators from all 3 services in the United Kingdom were converted using this 
Control Trainer System at the Army School of Ammunition. Anite Systems Ltd. designed the system, initially for the 
wheelbarrow MK8+, which is an export model. 

System Description And Operation 

The system was designed primarily to enable operators to become familiar with the functionality of the Hand Controller. 
To do this we have four control work stations. Each station comprises a standard desktop PC with P75 processor, a 15” 
monitor, keyboard, mouse and joystick. A standard hand control unit attaches to the processor via an interface box. The 
software provides 3 separate, but selectable 2D VR environments in which to train. In each training scene a graphical 
representation of the Wheelbarrow is produced which responds to commands from the Hand Controller with the ability to 
‘fire’ standard IEDD weapons at simulated targets in the same way as the real vehicle. The ‘F’ keys on the keyboard in 
conjunction with the joystick allow the operator to move around the scene and also to obtain the classic ‘through the 
camera’ view he would get with the Wheelbarrow. The joystick control is independent of the Hand Controller and has no 
effect on the movement of the vehicle. In each scene there are a number of configuration facilities to add a challenge and 
variety to the training task. These are: 

Street Scene 

• 3 incident control point positions 
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• 4 IED types (attache case, petrol bomb, package and car) 

• 3 pre-set IED locations (telephone box, under car and car trunk) 

• Manual IED positioning 

• 3 drive camera positions 

House Scene 

• 3 drive camera positions 

• 3 IED types (petrol can, bottle and attache case) 

• 5 IED locations (lounge, study, kitchen, bathroom, bedroom) 

• Manual IED positioning 

• 3 lounge layouts 

• Manual furniture positioning 

• Door set-up 

The Test Track adds a competitive edge to the system and comprises an obstacle course designed to test the operator’s 
control of the vehicle. The obstacles include: 

• Railway track 

• Stairs 

• Uneven roadway 

• Steep slopes 

• Narrow corridors 

• Traffic cones 

This is a timed task and a clock begins when the Wheelbarrow crosses the start line. If he should hit a cone it adds 30 
seconds to the time. He can not drive through walls to take short cuts! 

Benefits To Date 

This was the first computer based training system we had into service before the main hardware. There have been a 
number of obvious advantages so far: 

(1) It enabled the MK8B to enter service in a timely and efficient manner with personnel already trained in its use. 

(2) It is now a fully established training system, which we use for all new operators who come on courses. It allows 
repetitive training in a controlled and comfortable environment without the need for heavy instructor commitment, 
equipment support or a training area. 

(3) It has significantly reduced the repair bill to the vehicle fleet. 

Future Development 

Experience with the Control Trainer System has recognised its limitations. For what it was designed to do it does it very 
well. There are, however, a number of potential improvements to the system that we have discussed with Anite. 

• Improved graphical environment 

• A configurable scene generator 

• A scenario generator 

• The ability to use the full range of IEDD weaponry 

• A faults generator 

These modifications would provide a much greater variety of more realistic situations and give students a more flexible 
and testing training system. Ultimately our aspiration is to make this part of a family of a team training system. The 
prototype system for the ANDROS RCV has many parallels in its concept. There is certainly scope for an exchange of 
ideas. 
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IEDD THREAT ASSESSMENT TRAINER 

The second computer based training system is the IEDD threat assessment trainer. This is currently a concept 
demonstrator developed by Westland System Assessment Ltd. in conjunction with Centre for Human Sciences and, 
unlike the RCV Trainer, concentrates on the ‘soft skills’ of the IEDD operator. 

In any IEDD situation the critical time is generally considered to be the first 10 minutes after arrival at the scene. In a 
very short space of time the operator must be briefed, establish the safety of the ICP, obtain information through 
questioning, make an initial assessment of the threat and devise his initial EOD plan and response. 

This is no easy task and experience has shown that many operators find this aspect of the task the most difficult to 
master. We are currently developing a concept demonstrator, which replicates this process and is based on an existing 
classroom exercise. 

System Description And Operation 

The current system, which has undergone a number of developments, is based on a desktop multi-medium PC running a 
P90 processor with 16 MB of RAM and a space mouse. The visualisation software is Superscape Visualiser 4.00 running 
under MS DOS 6.22. MS-Windows 3.11 is also loaded to run MS-Write for exercise feedback. 

The Exercise 

On beginning the exercise the student is given the opportunity of working through a tutorial. This is simply designed to 
teach the student the basic computer skills necessary to use the trainer and an explanation and practice exercise with the 
virtual world space mouse. All students are required to complete the tutorial at least once before the threat assessment 
exercise. 

On completion of the tutorial the student is presented with a screen which basically sets the scene. He is then given an 
audio brief that tells him exactly what has happened. The imaginary scenario used is an explosion in Canterbury in 
southeast England, in which there are a number of casualties. He has been tasked to deal with the problem and arrives at 
the scene within 15 minutes. As the brief is given, he is presented with a schematic diagram of the scene and once this is 
completed he is presented with an initial view of the scene from the police control point. His exercise now begins. 

To the right and bottom of the screen there are a number of user interaction icons coloured blue when non-active and 
green once selected. There are also keyboard shortcuts for each icon: 


• Schematic map 

• Question and Answer 

• Vewpoints 

• Visualisation 

• Decision 

The Assessment Session is his final choice. Once he has sufficient information he selects the assessment icon which then 
questions him on his understanding of the situation. He will answer about 30 questions, most of which only require a 
mouse click in response. A few will require a typed entry. 

On completion of the assessment session, he is presented with a final screen that allows him to walk through the virtual 
world highlighting important aspects such as cordon positions and security cameras in the area. The second choice is 
feedback. This displays messages, which will have been triggered by certain selections made in the assessment session, 
informing where the student has gone wrong or congratulating him on making the correct choices. 

Benefits Of An IEDD Threat Assessment Trainer 

This aspect of an IEDD task is the most critical. If an operator is able to make an accurate assessment of the problem he 
is then faced with his render safe plan, which should follow quite logically. His skills with the equipment can only be 
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perfected by practical training, but we see this as a pre-cursor, in many respects, to his practical training. This interactive 
demonstrator uses a relatively uncomplicated scenario. There are of course many real situations that have occurred which 
are far more complex and testing. It follows the methodology of dealing with an IED task taught in the classroom and is a 
natural extension to this teaching. 

FUTURE DEVELOPMENTS 

This is only the first hurdle. With an improved graphics environment, a greater selection of scenarios and voice-activated 
commands instead of keystrokes, we are certain this will become a valuable training tool. It will never replace reality, but 
it will help develop our professional EOD operators and is another step towards our ultimate team training system. 



19-1 


AN INTELLIGENT TUTORING SIMULATION FOR MILITARY 
OPERATIONS IN URBAN TERRAIN 

John Leddo, Ph.D. and James Kolodziej 
Research Development Corp. 

2875 Towerview Road 
Suite A4 

Herndon, VA 22071-3205, USA 

Simulation has played a major role in military training. Distributed Interactive Simulation (DIS) allows multiple trainees 
to interact in real time on a common training problem. While DIS is a powerful training tool, a trainer is typically 
required to review trainee performance and make the appropriate teaching and remedial points. As training scales to 
larger and larger exercises, the trainer will naturally focus on general team performance at the expense of individual 
training needs. Intelligent tutoring systems (ITSs) have focused on providing instruction on a one-to-one basis. 
Integrating DIS and ITS technologies offer the opportunity to capitalize on the strengths of both: the ability to conduct 
large scale team exercises while providing each trainee with personalized instruction. The present paper reports a Phase 
II Small Business Innovation Research (SBIR) project, sponsored by the U.S. Army Simulation, Training and 
Instrumentation Command (STRICOM) in which an Intelligent Tutoring Simulation™ was developed to tram Army 
Infantry squad and fire team leaders the skills they need to cooperatively perform military operations in urban terrain 
(MOUT). 

BACKGROUND 

Simulation has long played a significant role in military training. Whether live, virtual or constructive, simulation is the 
primary mechanism by which soldiers receive practical training or conduct analysis to prepare for potential missions that 
they may be called upon to accomplish. 

As technology has improved, emphasis has been placed on creating simulators with higher fidelity and networking 
simulators to support team problem solving. In the military, this is best exemplified by the creation of distributed 
interactive simulation (DIS). DIS supports training by allowing large numbers of trainees to work together in order to 
collaboratively solve a problem as they would in a real setting. This is perhaps the most critical innovation within the 
field of simulation as virtually all military problem solving exercises are collaborative activities. As technology 
improves, one can only expect to see a capability to provide even more comprehensive training exercises with greater 
numbers of participants. 

As technology has increasingly enabled training exercises to increasingly approximate realism by means of DIS-based 
virtual simulations, a subtle tradeoff is being made. On the one hand, the increased realism provides greater transfer of 
problem solving skills to the real world tasks a soldier will perform. On the other hand, as simulation substitutes for 
instructor-based training, the learning benefits of having active assessment and feedback by a qualified instructor is 
sacrificed. 

The present paper addresses a project we have worked on that tries to give the training community the best of both 
worlds: distributed interactive simulations for realistic team training and individualized instruction for each soldier based 
on his learning needs. To accomplish this, integrated intelligent tutoring system technology has been integrated with 
simulation technology. 

Intelligent tutoring systems (cf., Bma, Ohlsson and Pain, 1993; Greer, 1995) use artificial intelligence to place teaching 
mechanisms into a training system. An intelligent tutoring system (ITS) is defined by three characteristics: an expert 
model of how to solve a problem correctly, a model of what the trainee knows that can be compared to the expert model, 
and a pedagogic model that tells the system how to teach the trainee what an expert knows given what the trainee knows. 

Each of these components is missing from typical technology-based simulation environments. For this reason a typical 
simulator can provide practice but not instruction. The contention of the present paper is by integrating the approaches 
offered by the ITS and simulation communities, an enhanced paradigm of realistic scenarios, coupled with the 
instructional benefits of ITS technology can be created. We term this technology “intelligent tutoring simulation”. 
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In order to create a successful intelligent tutoring simulation environment, several technical hurdles must be overcome. 
This is so because the strengths of each technology are problematic for the other. 

For example, simulation environments typically support open-ended behaviors. A trainee can execute any action 
allowable within the simulator at any time, regardless of whether or not the action is prudent. An ITS, on the other hand, 
is trying to assess the student. Because it is much easier for a computer to process well-defined linear event sequences, 
most ITS work has focused on domains where well-defined linear procedures exist, such as algebra and basic computer 
programming. 

In a simulation environment, there need not be a predetermined “correct” answer to the scenario being trained on. 
Indeed, in many real world problems, a “correct” answer may not be known. In such cases, the instructor tries to 
evaluate the quality of solution process rather than the outcome of that process. In a military problem this could include 
such things as proper use of reserves, establishing supply lines, etc. In the ITS world, the emphasis has been on domains 
where an objective “right answer” exists (e.g., mathematics) as computers are much better at dealing with well defined 
problems with computable solutions than they are at dealing with ill-defined problems with no computable solution. 

As noted earlier, the heart of DIS is collaborative problem solving. However, the goal of an ITS is to assess and teach 
each student based on his individual needs. Because it is easier to do this process when only a single student is being 
trained, ITS researchers have focused on single student ITSs. 

The present paper addresses how these technical issues were reconciled so that technology could be developed that 
supports open-ended team problem solving behaviors while providing each team member with individual instruction. In 
addition to these technical issues, there were other issues that relate to both the simulation and ITS communities. 

One of the major weaknesses in the ITS and simulation fields is that both technologies are costly to develop. They 
typically require highly skilled professionals to develop. Once developed they tend to be highly inflexible such that they 
often become quickly outdated and need to be rebuilt from scratch when updated. When such rebuilding occurs, there 
tends to be little reuse of the previous system’s technology in the newer version. 

An argument can be made that if intelligent tutoring simulation technology is to have a future, it must be made adaptable 
and generic enough so that the technology can be constructed cost-effectively, be modifiable and updatable by the user, 
and have considerable reuse across projects to maximize the value to the user for the dollars invested. 

These additional objectives influenced the design of the present technology (which is discussed in the next section). The 
effectiveness of the present technology was demonstrated by building generic and modifiable technology that was 
extended as the project progressed and was transferred to another project. 

THE INTELLIGENT TUTORING SIMULATION TECHNOLOGY 

The present technology was developed in partnership with both the Directorate of Operations and Training (DOT) and 
the Battle Lab of the Ft. Benning Infantry community. We are particularly grateful to Mr. David Reiss who was our 
point of contact at Ft. Benning and who provided us with support in the form of subject matter experts and project 
reviews. 

In working with the Infantry community, the Military Operations in Urban Terrain (MOUT) task of clearing a building 
was identified as a high training priority. A decision was made to focus on leadership training, first at the fire team 
leader level and then at the squad level. 

In both cases, leaders perform two major functions: first, they are responsible for making decisions and directing the 
actions taken by their respective units; second, they are responsible for correcting mistakes made by members of the unit. 
Therefore, the training objectives that were established for the present technology had both of these responsibilities in 
mind. 

Part of an ITS is the expert model of how to solve problems in the domain. Additionally, a simulation requires a model 
for computing events within the domain. Both of these are clearly linked. The simulator needs to record trainee actions 
to compute how events should unfold in the simulation. The ITS needs to record trainee actions to determine learning 
needs. 
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In order to build the domain model and the expert problem solving model extensive knowledge engineering was 
conducted with subject matter experts supplied by Ft. Benning. This was supplemented by observing live training 
exercises at the McKenna MOUT Site at Ft. Benning, conducting role playing exercises at RDC headquarters and by 
reading published doctrine. 

These knowledge models formed the basis of the training technology. The next step was to build the actual system. 
This was done in phases. In the first year of our two year project, a one person trainer for the fire team leader was built. 
In the second year of the project, a second, networked trainer for the squad leader was added. 

In developing the trainer for the fire team leader, the goal was to create an architecture that would be later scaleable to 
adding the squad leader. A second goal was for the technology to be as generic as possible so as to be modifiable when 
necessary and usable with simulators other than the one created for the present project. 

In order to meet these objectives, a simulation-based intelligent tutoring system was constructed that had three 
components. The first was the MOUT simulator that allows a trainee to assume the role of a fire team leader and direct a 
four man fire team in the task of clearing a building. The floor plan used in the simulator is modeled after a building 
within the McKenna MOUT site. ^ 

The simulator creates virtual (but non-immersive) simulations of the inside of a building. World Toolkit™ was used to 
create these simulations, which run on a standard Pentium PC. 

The trainee assumes the role of the fire team leader and directs his team by means of key strokes. Once the key stroke 
command is issued, movement is automatic. The fire team carries out the commands issued by the trainee, regardless of 
whether they are “correct”. Occasionally, for pedagogic purposes, a fire team member carries out a procedure 
incorrectly. At this point the trainee has the opportunity to correct the mistake, again using keyboard commands. If the 
trainee does make the correction, the mistake is not repeated. If s/he does not, then the system continues to cause the fire 
team member to repeat the mistake. 

The second component of the system is the intelligent tutor. The intelligent tutor is responsible for assessing the trainee 
actions in the simulator, determining whether corrective instruction is needed and directing the simulator to provide such 
instruction when necessary. 

As stated earlier, an intelligent tutoring system is comprised of three components: an expert problem solving model, a 
student model and a pedagogic model. The logic for how these components were constructed is based on the project 
objectives. First, one of the major challenges to integrating ITS technology with simulation technology is to allow the 
ITS to support the open-ended behaviors and ill-structured problems found in many simulation environments. Second, it 
is desirable to have generic technology that is, in principle, reusable for other training applications and not tied to the 
specific simulator or set of scenarios that simulator might run. 

In order to accomplish these objectives, two innovations were made to the typical expert model found in many ITSs. 
First, in order to handle the rich MOUT problem solving domain and allow more open-ended behaviors, a richer 
knowledge representation framework than is typically found in many production rule-based expert systems was used. 
This knowledge model framework was based on previous empirical research (Leddo, Cohen, O’Connor, Bresnick, & 
Mann, 1990) on how experts solve problems. 

This research shows that experts use a variety of problem solving approaches that are richer than a simple production rule 
or other single formalism process. As a result, an Integrated Knowledge Structure (or INKS) framework was created that 
blends scripts, production rules, semantic knowledge and mental models into a single formalism. INKS allows an expert 
model to process known problem solving sequences as a production rule system would, but also allows a system to use a 
mental model to reason from first principles given the semantic information available in a situation. 

For example, when a fire team moves down a hallway and approaches a door, there is a fairly routinized procedure for 
stacking, breaching the door and entering the room. In cases where multiple doors are present a decision must be made 
as to which room to clear first. In such cases, at least two decision making processes are possible. The first is to 
“hardcode” every possible permutation of how many doors there are, whether they are marked or unmarked (indicating 
that they are already cleared) and whether the doors are open or unopened (indicating a potential threat as an open door 
constitutes a potential line of fire from enemies within the room). The expert system could literally evaluate each of the 
antecedent conditions to determine which rule to fire and in doing so, which room to clear. 
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An alternative approach, which the INKS expert model allows is to reason from first principles. To accomplish this, the 
INKS knows about the goal of preserving the safety of the fire team. It knows that being in an enemy line of fire 
constitutes a safety threat. Therefore, when confronted with multiple rooms, the INKS can evaluate each room to 
determine which constitutes the greatest safety threat and then decide to clear that one first. By using this mental model 
approach to evaluating safety threats, the system is not required to have a preset rule to assess a trainee’s decision, but 
can still make the assessment based on known goals and situational (semantic) features. 

Being able to reason from first principles leads into the second objective discussed, namely creating an ITS that is 
generic. Having mental models that reason from first principles is a step in the right direction as this supports reasoning 
about general cases rather than hardcoded examples. 

This train of thought was continued by building the INKS to be a generalized MOUT expert rather than one that was 
knowledgeable about the specific simulator, floor plan or scenarios used. This was accomplished by encoding the 
domain knowledge in generic terms such as moving down types of hallways rather than specific hallways, entering types 
of rooms rather than specific rooms, etc. By doing so, the system is able to reason about any scenario as long as it can 
make the determination about what type of situation it is in. This makes the ITS independent of the simulator. 

A good analogy is be a human trainer. A human trainer is a generic domain expert. S/he can be brought into any training 
environment and act as a trainer because s/he knows about the domain and can apply that knowledge to the specifics of 
the training environment because s/he can get, by observation, the relevant information about what learning objectives 
are being taught and what scenarios are being used to teach them. Before discussing how the ITS is getting this scenario 
information, the other two components of the ITS are briefly discussed: the student model and the pedagogic model. 

The student model uses the same INKS used in the expert problem solving model. In this way, student actions are 
compared directly to the expert model and assessed accordingly. 

For the pedagogic model, there are many options as there is no “one right way” to teach any subject area. Case-based 
reasoning was chosen as the pedagogic model for two reasons. First, it lends itself well to the scenario-based instruction 
that the simulation would use. Second, it supports individualized instruction as the student learning style itself can be 
treated as a case to which remedial instruction could be matched. In the present system, four forms of remedial 
instruction are used: showing trainees the consequences of their actions with logical scenario outcomes (e.g., a trainee 
fails to correct a fire team member action that places the fire team in danger, so the case-based reasoner causes a soldier 
to be killed by an enemy.); providing trainees with miniature scenarios to practice a faulty skill; providing trainees with 
an auditory explanation of what they did wrong; and providing trainees with a text-based explanation of what they did 
wrong. The case-based reasoner (CBR) cycles through these potential forms of remediation until it finds a form by 
which the trainee does not repeat the mistake being remediated. When this occurs, the CBR infers that it has matched the 
trainee’s learning style. 

Having discussed the simulator and the ITS, the third component of the technology is addressed. A generic INKS was 
created to serve as the expert problem solving model. This INKS operates as a human trainer does, inferring from the 
events in the simulation what type of scenario is being trained so that it can assess the trainee and determine whether 
remedial instruction is needed. 

In order to accomplish this, real time feedback is needed as to what events are happening in the simulation and what 
actions are being taken by the trainee. To accomplish this, middleware, which is the third component of our technology, 
was created. 

The middleware has two major components. First, in order to provide the INKS with information regarding events in the 
simulator, a semantic overlay of the floor plan used in the simulation was created. This semantic overlay contains such 
information as where the doors are, whether they are open or closed, how many locks and hinges they have, etc. 
Therefore, as the trainee moves through the simulation, the middleware can continually pass this information to the 
INKS. In essence the middleware acts as the “eyes and ears” of the INKS so that it can evaluate what type of situation 
the trainee is in and then use its expert model to determine appropriate actions the trainee should take. When the trainee 
does take an action, the middleware also passes this information to the INKS so that it can assess the trainee’s action 
against its expert model. 

If the trainee’s action matches the expert model, then no remediation is provided. If remediation is needed, the CBR 
generates a requirement. This is passed to the middleware. Here is where the second major component of the 



19-5 


middleware comes in. The middleware also contains information about simulator primitives so that it can cause the 
simulator to produce necessary remediation. The basic simulation-based ITS architecture is illustrated in Figure 1 below. 

In year 2 of the project, the goal was to add the squad leader to the training technology. As was the case in the first year 
of the project, the first step was to conduct knowledge engineering with domain experts, supplemented by published 
doctrine to gather the necessary domain knowledge. 



Figure 1. Basic ITS Architecture 

However, the goal in year 2 was more than to simply build a second simulation-based ITS. The goal was to create a team 
trainer. Here, the challenges of creating a distributed problem solving environment while still preserving individualized 
instruction needed to be addressed. There was an additional technical challenge. The project goal was to create an 
Intelligent Tutoring Simulation for a squad leader and a fire team leader working together. However, a squad has two 
fire teams. Therefore, in order to preserve the realism of a two fire team squad, intelligent agent technology was used to 
play the role of the second fire team (this technology is discussed shortly). 

Fortunately, the basic year 1 architecture supported these extensions. First, the MOUT simulator was reused in year 2 
(the floor plan was extended to provide a starting hallway for the squad). In order to give the squad leader a separate 
perspective corresponding to what he would see, a duplicate simulator but with a camera (viewpoint) corresponding to 
what he would see was created. The squad leader was also provided with a sky view so that he can still watch the actions 
of the fire team after they disappear into a room. Finally, while the fire teams move as a unit, the squad leader is not 
tethered to either fire team but moves himself manually using key strokes. However, when he directs the movement of 
one of his fire teams, their movement is automatic as was the movement of the fire teams when directed by the fire team 
leader in the year 1 system. 

The next step was to build the appropriate ITS for the squad leader. The key step was to build the squad leader expert 
model. This was done in the same format as the fire team INKS. The CBR was also updated to include the remedial 
instruction for the squad leader errors. By creating separate tutors for the fire team and squad leaders, an environment 
was created where each trainee had his own tutor that was responsible for providing him with personalized instruction 
based on his learning needs. 

Once the simulator with the squad leader perspective and the squad leader ITS were created, middleware was constructed 
to link the two as was done in the fire team leader trainer. This middleware was essentially the same as for the fire team 
leader. The semantic overlay of the floor plan was identical. The main difference was to be able to pass the squad leader 
commands in the simulator to the squad leader ITS. 

The technology, as described above, constitutes two separate ITSs, even though the second one was constructed far more 
cheaply than the first. However, the goal was to develop a team training environment so the two systems had to be 
linked. The linkage was provided through the middleware. Here, in each system, the middleware passed each trainee’s 
simulator commands not only to his own ITS but to the other trainee’s middleware. The other middleware then updated 
the second simulator so that the trainee would view the same events as his partner. To synchronize the two simulators, 
an internal simulation clock was created. Each event that was passed between the two middlewares was timestamped so 
that the receiving middleware could update its simulator in a way that would preserve the synchronization between the 
simulators. The synchronization between simulators is a standard DIS problem and the present paper does not claim to 
have made any innovation in this area. 

The resulting technology was two Pentium PC computers that used a point to point connection. One computer had the 
squad leader version of the technology, while the other had the fire team leader version. Figure 2 below shows the 
architecture for the two person intelligent tutoring simulation. 
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Figure 2. 2 Person Intelligent Tutoring Simulation Architecture 

There is one final issue. The present technology is comprised of one squad leader trainer and one fire team leader trainer. 
However, a full squad has two fire teams. The second fire team was “played” by an intelligent agent. 

One of the features of the present ITS technology is the expert problem solving model. This expert problem solving 
model evaluates the trainee by computing its own solution to the problem and evaluating the trainee against that solution. 
This feature of the expert model was used in order to create an intelligent agent that would do the same. 

Therefore, when the squad leader issued a command that would ordinarily be carried out by the second fire team (fire 
team B), the expert model would generate an expected action on the part of that fire team. These commands were then 
automatically carried out. This enabled the agent for fire team B to respond to squad leader commands. 

There were cases where the fire team leader would normally issue his own commands. In this case, the expert model 
would be receiving information, via the middleware, of what events were happening in the simulation. The expert model 
then computes what the fire team leader should command. In this case, rather than waiting for the trainee to issue a 
command, the expert model simply issues the command itself. 

This ability for the expert model to operate either in assessment mode (when a real trainee is issuing commands) or in 
agent mode (to issue a command itself) created a unique feature of the technology. Specifically, the intelligent tutoring 
simulation could not only support two person training, but also single person training in either a fire team or squad leader 
role. This was accomplished by having a toggle that transferred control of the fire team leader or squad leader from 
human to computer and back again. 

DEMONSTRATION OF THE ROBUSTNESS OF THE TECHNOLOGY 

A principal goal of the present project was to create a generic architecture that could be a model for rapid technology 
construction and software reuse. While a demonstration of this was not part of the original project plan, there was a 
serendipitous opportunity to provide such a demonstration when RDC received a Phase I Small Business Technology 
Transfer contract with the Army Research Institute to develop a virtual schoolhouse. 

In this project, the goal was to transfer intelligent tutoring simulation technology from the present project and intelligent 
agent technology from a DARPA-sponsored project to develop team training that could be delivered over a network such 
as the Internet. 

Because higher fidelity simulations require higher bandwidth to be transmitted over a network than lower fidelity ones, a 
decision was made to use a low fidelity simulation for that project. The entire functionality of the present project was 
duplicated in the form of a networked, low fidelity, constructive simulation for the ARI project. To accomplish this, both 
squad and fire team INKS’ were reused in their entirety and enhanced them and the middleware was reused in its 
entirety. This was done by creating a constructive version of the same virtual simulation such that the primitives and 
semantic overlay provided in the middleware also mapped to the constructive simulation as well as the virtual one. 

Essentially, then, the present project technology was completely reused and even enhanced (additional knowledge was 
added to the INKS in this second project), except for the virtual simulator. However, between the two projects it was 
demonstrated that the same ITS technology could be applied to two simulators (a constructive and virtual one) without 
changing either the ITS or the middleware and the same simulator could be applied to two different ITSs (a fire team 
leader and a squad leader) without changing the simulator (except for the camera angle) or the middleware. 

There was a further test of the robustness of the present technology. A scenario editor was created that allowed a user to 
enter his own scenarios by manipulating certain parameters (e.g., number and location of enemies, their level of training 
and whether they were combatants, number and location of civilians). As stated earlier, the present technology allows 
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the computer to run the role of the fire team leader or the squad leader because it processed scenario information in real 
time and made its own decisions regarding what actions to take. 

In this case, numerous demonstrations of our technology were given. Each time, observers were allowed to create their 
own scenarios and have the expert model run the simulation. There were no cases where the scenario created by an 
observer “broke” the system or created an unexpected event. This demonstrates the robustness of our expert knowledge 
model, which is the heart of our technology. 

It should be noted again that the virtual schoolhouse system completely duplicated the functionality of the present 
system. With the addition of the scenario editor, it actually had an additional feature that the present technology does not 
have. What makes this accomplishment even more noteworthy is that the virtual schoolhouse project was developed 
using Phase I level resources which are about one-sixth that of a Phase II development effort. This demonstrates how 
such technology, if carefully constructed, can help leverage future development efforts such that the technology can be 
replicated and enhanced at considerable savings. 

CONCLUSIONS 

The present project demonstrates the feasibility of integrating intelligent tutoring system and simulation technologies. It 
is argued that the value of such integration is by maintaining the team training spirit of today’s DIS technology, while 
providing each participating soldier with the opportunity to receive individualized instruction based on his learning 
needs. 

Further, the present project demonstrated an approach for creating such technology that makes it extendible and reusable 
at a fraction of the original development costs. This can be considered an important feature as it suggests that such 
functionality could be added to today’s (and tomorrow’s) simulator development efforts at a modest cost. 

The goal in this project, then, has been to demonstrate the enhancements that could be made to DIS technology by 
integrating ITS technology and that these enhancements could be done in a cost-effective manner. By achieving these 
goals, it is hoped that a paradigm shift can be effected in the way simulation-based training is conducted. The authors’ 
vision is that future training technology will be a marriage of simulation and intelligent tutoring systems, creating 
distributed interactive intelligent tutoring simulations. 

REFERENCES 

Brna, Ohlsson, S., & Pain, H. (1993) (Eds.) Proceeding of Artificial Intelligent in Education ‘93. Charlottesville, 
VA: Association for the Advancement of Computing in Education. 

Greer, J. (1995) (Ed.) Proceeding of Artificial Intelligent in Education ‘95. Charlottesville, VA: Association for 
the Advancement of Computing in Education. 

Leddo, J., Cohen, M.S., O’Connor, M.F., Bresnick, T.A., and Marvin, F.F. (1990). Integrated knowledge elicitation 
and representation framework (Technical Report 90-3). Reston, VA: Decision Science Consortium, Inc. 





20-1 


TELEPRESENCE AND INTERVENTION ROBOTICS 

Nathalie Cislo 

Laboratoire de Robotique de Paris 
10-12, Avenue de V Europe 
78140 Velizy-Villacoublay, FRANCE 
cislo@robot.uvsq.fr 


ABSTRACT 

In the field of Mobile Robotics applications dedicated to inspection or intervention in hostile, unreachable or 
unstructured environments, human operators are “processed” in the control loop developed for a Telepresence System. 
On the one hand dynamic situations suffer from a lack of automation degree by mobile robots but on the other hand a 
complete task robotisation can run counter to economic or human constraints. In mine clearance activity for instance, 
mobile robots and especially teleoperated semi-legged robots can be seen as a solution, not to replace the mine clearance 
specialists, but as a safe tool for human operators in some well-defined situations. 

As human interaction with a machine is often oriented towards application, the teleoperation of a mobile robot under 
geometrical constraints is not easy to achieve if no training period occurred previously for the operator. A footbridge 
exists between training and relevant missions on navigation tasks: a real activity with a mobile robot can be prepared in a 
micro-world — a virtual world or an Augmented Reality world. 

But the use of Telepresence systems, presented here as destined for an inescapable expansion because of cultural 
yearning with ancestral roots, is submitted to human factors. In this paper, after a discussion about the ubiquity myth in 
Telepresence, we present a Telelocomotion system with a Control-Command-Communication (C3) strategy adapted in a 
Service and Intervention context to semi-legged robots like our RAMSES (French acronym for Mobile Autonomous 
Robot with an Advanced Support System). The vision aspect is focused on and vision sickness issues generated by the 
motion of visual feedback are analysed. Some solutions to avoid or diminish troubles with either an appropriate camera 
or specific control laws are presented. Experiments are in progress to study the relevance in Telelocomotion of a 
Behavioural Transform by means of gestures produced by a dedicated optical fibre glove to operate a legged robot in 
comparison with the use of a joystick. 

INTRODUCTION 

In the field of Mobile Robotics applications dedicated to inspection or intervention in hostile, unreachable or 
unstructured environments, human operators are “processed” in the control loop developed for a Telepresence System. 
On the one hand, dynamic situations suffer from a lack of automation degree by mobile robots, but on the other hand, a 
complete task robotisation can run counter to economic or human constraints. In mine clearance activity for instance, 
mobile robots and especially teleoperated semi-legged robots can be seen as a solution, not to replace the mine clearance 
specialists but as a safe tool for human operators in some well-defined situations. 

Different investigations are possible when dealing with legged robots and dedicated tasks. At the L.R.P. (Laboratoire de 
Robotique de Paris), we focus our attention on problems related to the stability of quadruped robots with a "force control 
and distribution" approach [Nak.89] [Kim.90]. Currently we study legged robots with respect to various situations 
(different topologies, rough terrain, partially unknown environments). Such an aim requires matching the task to be 
accomplished and the mechanical specifications of the robot. We have observed in many cases that hybrid structures 
(e.g., legs plus wheels) [Vil.93] [Fon.94a] are very well adapted to navigation tasks in some ways like stability (i.e., 
teleoperation is facilitated if the general instability of the robot is overcome). 

As human interaction with a machine is often oriented towards application, the teleoperation of a mobile robot under 
geometrical constraints is not easy to achieve if no training period occurred previously for the operator. A footbridge 
exists between training and relevant missions on navigation tasks: a real activity with a mobile robot can be prepared in a 
micro-world - a virtual world or an Augmented Reality world. 


Paper presented at the RTO HFM Workshop on u The Capability of Virtual Reality to Meet Military 
Requirements ” held in Orlando, USA, 5-9 December 1997, and published in RTO MP-54. 
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But the use of Telepresence systems, presented here as destined for an inescapable expansion because of cultural 
yearning with ancestral roots, is submitted to human factors. In this paper, after a discussion in the first section about the 
ubiquity myth in Telepresence, we present in the second section a Telelocomotion system with a Control-Command- 
Communication (C3) strategy adapted in a Service and Intervention context to semi-legged robots like our RAMSES 
(French acronym for Mobile Autonomous Robot with an Advanced Support System). The vision aspect is focused on and 
vision sickness issues generated by the motion of the visual feedback are analysed. Some solutions to avoid or diminish 
troubles with either an appropriate camera or specific control laws are presented. Experiments are in progress to study the 
relevance in Telelocomotion of a Behavioural Transform by means of gestures produced by a dedicated optical fibre 
glove to operate a legged robot in comparison with the use of a joystick. These experiments are described in the third 
section. 

TELEPRESENCE 

Introduction 

In French language dictionaries the « Telepresence » neologism is not yet referenced. However, more and more 
researchers and manufacturers take an interest in Telepresence. This first section deals with ubiquity and myths 
becoming Virtual Reality in order to try to explain the inescapable expansion of Telepresence. 

Ubiquity 

« Ubiquity Society » 


According to Jean Cazeneuve, a contemporary French sociologist, we are a « ubiquity society » [Caz.70] [Alb.95], 
resting on audio-visual media, extraordinarily rapid diffusion of hertzian waves adding an almost instant-dimension to its 
universal nature. Does that mean that people daily apprehend media with ubiquity characteristics and that it is possible to 
be present in several places at the same time? 

In any case, this is the viewpoint of several contemporary researchers. Media like printed matter, painting reproductions, 
disks, and most of all telematic networks , and information highways , are said to allow ubiquity in works of art and 
information [Asc.97]. As early as 1928, Paul Valery, in a visionary text [Val.28], was considering « a sort of ubiquity » 
for works of Art, imagining a future home-distribution with techniques from a so-called « Sensitive Reality » without 
anything less than what is called today « Virtual Reality ». Moreover, there ar many e who consider that going on the air, 
on television, speaking on the phone, are many ways to have a certain type of ubiquity [Alb.95] [Bal.95]. 

But is it right to talk about ubiquity in these cases? 

« Corporal Ubiquity » and « Virtual Ubiquity » 

Most adult human beings have an image of themselves, from their bodies [Sch.68] [Cor.72], that causes them to 
understand that the type of ubiquity we designate as « corporal » is unrealistic: except in some primitive tribes and 
among some heautoscopic trouble-related illnesses, the mirror image is just an image of the proprioceptive body. Our 
human condition and our world perception do not permit considering ubiquity in any other way than the « coiporal 
ubiquity ». Now it is no more possible to be the same human being in several identical bodies, than to exist in several 
places at the same time, acting differently on the environment in different places (or in one place) like Marcel Ayme’s 
heroine in its short story Les Sabines [Aym. 43]. 

Going on the air, on television, speaking on the phone, is the option to be « virtually ubiquitous ». Only a voice « image » 
is transmitted quasi-instantaneously, then is presented in several places, though the direct voice and the real body that 
produces this voice exists in only one place. 

Mythology and Theology 

The « immemorial ubiquity dream » [Bal.95] of man, namely « corporal ubiquity », appeared in mythology and theology: 
the Brahman [Ind.95] [Sid.95] and Greek pantheons [Pyt.95] had the fanciful gift well before the monotheist religions 
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initiated the ubiquity dogma [Bre.95] [Dum.95]. If the pluri-presence 2 , and precisely « virtual ubiquity » can be 
considered for human beings thanks to brand new technologies like Virtual Reality, the pluri-existence is and will be 
reserved to gods: the Visnu avatars and the Trinity are a few examples. Jean-Paul Papin sums up the discussion as 
follows [Pap.97]: 


« Ubiquity is to presence what Trinity is to existence » 

Trinity deals with several beings being together, that « are » and « exist», though ubiquity deals with the same being, 
present in several places at the same time. 

Telepresence and «Quasi Corporal Ubiquity» 

Then, is it right to speak about ubiquity - meaning “corporal ubiquity” - in Telepresence? 

« Quasi Corporal Ubiquity » 

We have tried to demonstrate that « virtual ubiquity » is possible by means of a hardware and software interface. Then to 
be able to talk not only about Telepresence but also about « quasi corporal ubiquity » it is sufficient to have at one's 
disposal: 


■ A control-command-communication software and hardware architecture that allows one to have a distant image 
of oneself, which is mobile and controlled with a one-to-one relation, 

■ Sensorial feedback, sufficient to give the impression of immersion in a distant, real environment, because of the 
previous architecture, 

■ Physical action abilities in this environment. 

It is only in the situation where the operator action has an effect in a world where he is supposed to be, that Telepresence 
can be talked about. 

Robot Concept 

The mobile robot is an operator image as the operator himself is in God's own image in Judaeo-Christian tradition. It has 
its own future in case of operator/teleoperator interface malfunction. 

The robot concept dates back to ancestral desires. The Hephaistos god, in Greek mythology, built some obedient voice- 
operated « tripods » to help him with his forging work. It was only in the '20's that some «little artificial 
anthropomorphic beings perfectly obedient to their masters » were designed for the first time, designated by the term 
« robot », in a play called « R.U.R. » (Rossom’s Universal Robot) [Tsc.20][Coi.82], by the Czech Karel Tschapek. 

If the Oxford dictionary defines the robot as follows: « a mechanical apparatus resembling and doing the work of a 
human being », it is not well accepted even in the S 80’s because « no existing and useful robot is alike human » [Coi.82] 
at this time. Nowadays, the expansion of biped, quadruped or hybrid mobile robots validates the robot formal definition 
especially if it is brought together with the Sagittarius concept [Fon.95]. 

Sagittarius Concept 

A mythical creature illustrates the « quasi corporal ubiquity »: the Sagittarius. It is particularly interesting to point out 
that a human operator can be directly superimposed on a robot. For that, we need to see the human locomotion and the 
gripping actions as the exact replica of what can be done by the front part of an antic creature: a Sagittarius (Figure 1). 


2 Presence in many places at the same time 
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Figure L Morphologic Equivalence Between a Human, a Sagittarius and a Robot 

It is also referred to as the mythical creature, half a man and half a horse, called a Centaur. The Sagittarius’ humanoid 
upper human body is dedicated to gripping and sensitive functions, it holds a bow (a tool!). The locomotive lower body 
comes from a horse. 

A one-to-one relation exists between the operator (the Sagittarius) and the teleoperator (the robot). The rear parts of the 
robot and the Sagittarius tend to improve the teleoperator stability and are totally controlled by the front part: the rear of 
the horse corresponds to the robot part that follows the "Leader". In our case, the wheels placed at the rear provide the 
global stability of the system without the contribution of the human "virtual" rear legs. The leaders are in the front and 
are directly operated by the operator’s legs. 

TELELOCOMOTION SYSTEM 

Semi-Legged Robot 



Figure 2. Layout of RAMSES lib. 

We developed a mini semi-legged mobile robot called RAMSES lib based on RAMSES and RAMSES II presented in 
[Fon.94b][Cis.95]. This robot has front pneumatic legs and powered wheels at the rear for better support (wheels placed 
in Gordini way) (Figure 2). 

It is able to walk (and roll) in an automatic or teleoperated mode to achieve what is called Telelocomotion [Khe.95]. Tliis 
type of robot affords a quasi-unconditional stability and has been designed keeping in mind small obstacle avoidance 
under teleoperation routines. 
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It is necessary to teleoperate the robot with the most direct relation between the operator and the mobile robot in order to 
provide an ergonomic human-robot interface. For that, we need the one-to-one relation given by the Saggitarius concept 
(Figure 1). 


In order to facilitate the development of a demonstrator, two operator fingers are equipped, instead of his legs: the 
movement is produced by an optical fibre-based dataglove called LightGlove© (Figure 3) that we especially developed 
for this application, which covers the hand of the operator. The robot legs copy the operator fingers [Fon.94a] [Fon.94b] 
[Khe.95] [Fon.95] [Cis.95]. 


Serial Link 



LightGlove 


RAMSES 


Figure 3. Legs Control with the Fingers. 


Obviously, a very efficient way to teleoperate a system can be a Telepresence system which uses modem tools (e.g. 
Virtual Reality), and as far as possible direct relations between human abilities (physical and psychological) and the 
robot’s characteristics. 



Figure 4. RAMSES lib and the LightGlove© 

TELELOCOMOTION DEVICES 

During the teleoperation mode, the operator is fitted out with a two-finger LightGlove©. The robot speed is controlled 
directly by the finger flexion speed. For visual feedback, the operator is assisted by a method which keeps his hands free. 
The environment is re-created in conjunction with virtual cameras fixed to the robot or according to the choice of the 
operator. 

The robot direction is determined according to the yaw of the arm’s proximal part around a fixed (vertical) axis through 
the forearm. The command is eased by a special cradle, the main component of the telelocomotion device presented in 
[Fon.95]. However, it may appear that the ergonomic constraints taken into consideration for the development of this 
device would not be sufficient to assure a global physico-psychological comfort to the operator. 
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VISION SYSTEM FOR TELELOCOMOTION 

D. Emadotte and S. Neuman [Ern.94] asserts that, during Telepresence, « it is important to have a general view of a 
scene and more precise views for manipulations ». From the two types of tasks to execute ( navigation and mission ), we 
classified two sorts of video systems embedded in the telelocomotion system [Cis.96]: 

■ A long distance video system, allowing the operator to locate the robot in its environment, 

■ A precise video system, efficient at a short distance, for sharp missions (inspection, obstacle avoidance, fine 
object manipulation with a gripping device). 

D. McGovern [Gov.93] gives experimental results about several types of video equipment on mobile wheeled robots with 
different architectures, on very unstructured grounds with obstacles: positives (bumps) or negatives (holes). 

To determine the appropriate video system for each of the two types of tasks mentioned above, we studied [Gov.93] and 
synthesised the results of experiments about video systems on teleoperated all-terrain wheeled robots. 

Navigation 

In navigation, our concern is with robots that can be piloted: 

■ Out of visual reach, 

■ Or into the operator’s visual field. 

The out of visual reach Telelocomotion requires an embedded video system. It is called « inside-out» teleoperation 
[Gov.95] with an « eye of snail » vision [Pap.95]. 

But when the mobile robot is in the operator’s visual field, the pilot sequences may be achieved: 

■ Either directly, with an “eye of God” view [Pap.95], 

■ Or in the same conditions as in out of visual reach, only with the “eye of snail” view, 

■ Or semi-directly by means of a video system located on another robot we called “relay robot”. In the mine 
clearance activity, robots of different sizes and velocities step in: a main robot can be accompanied by little, simple 
and very quick robots it supervises, whereas the others examine closely the suspicious objects. 

In the case of a unique semi-legged robot to teleoperate the solution we propose for navigation is an embedded video 
system. In order to allow the location of the robot in its environment, it is necessary to have a periscope type device with 
a 360° visual field. 

However, additional direct or semi-direct information does not have to be excluded when available. The visual 
information fusion, performed by the operator, can avoid some incidents listed by [Gov.93]: 

■ Roll-overs in « inside-out teleoperation », 

■ Frequent collisions with obstacles in “outside-in teleoperation”. 

During navigation, a black and white camera can be sufficient. However, it can be useful to discriminate surfaces merged 
in black and white (a road and its sides for instance), and therefore is preferred a colour camera. 

It must be noted that with a quick robot, rounding bends is easier when a camera rotation allows anticipating the curve to 
follow [Per.94]. This anticipation is feasible with a camera directional control on: 


Either the glance direction [Per.94], 
Or the legs’ direction. 
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Mission 

In the case of a video system dedicated to missions, it is interesting that colour is preferred for better detection of objects 
and obstacles. 

D. Ernadotte et al. [Em.94] conclude from a series of experiments that automatic systems of convergence and autofocus 
must be added on stereoscopic visions. We think that these devices are appropriate, but we stress the fact that they must 
be disconnectable by the operator during the task. 


VISION SICKNESS 

A vision system, even well-adapted to the task to execute, can generate troubles if it is placed on a mobile platform. 

From previous studies, and especially from [Oma.93], it appears that: « "Motion Sickness" is the general term describing 
a group of common nausea syndromes originally attributed to motion-induced cerebral ischemia, stimulation of 
abdominal organ afferent, or over-stimulation of the vestibular organs of the inner ear. Sea-, car- and air-sickness are the 
most commonly experienced examples. However, the discovery of other variants such as Cinerama-, flight simulator-, 
spectacle-, and space- sickness in which the physical motion of the head and body is normal or absent has led to a 
succession of "sensory conflict" theories which offer a more comprehensive etiologic perspective. Implicit in the conflict 
theory is the hypothesis that neural and/or humoral signals somehow traverse to other centres mediating sickness 
symptoms. » 

We propose to reduce the general term Motion Sickness to Vision Sickness [Cis.96] when it deals with symptoms due to a 
sensory conflict between visual stimuli from a picture in motion (provided for instance by a real or virtual scene on a 
screen) and a normal (or absence of) kinaesthetic feeling. Vision Sickness may then accompany every system with real or 
virtual visual feedback. 

In our telepresence system, visual feedback can provide vision sickness in three ways, corresponding to the three degrees 
of freedom (DOF) of a vision system placed on the platform of a semi-legged robot: pitch, roll and yaw. 

Pitch Sickness 

The most obvious sickness is associated with the pitch generated by the robot pneumatic leg height variations during a 
walking cycle [Khe.95]. At high velocity, the fuzzy and jerky visual feedback is as unusable as uncomfortable. 

Roll Sickness 

Though the roll is suppressed by the wheels at the rear on a flat terrain, it still exists on a very unstructured ground on 
which our robot is mainly dedicated to be teleoperated. This DOF may independently provide this type of discomfort. 

Yaw Sickness 

The last (but not the least) DOF sickness may be generated by a mismatch between the video system yaw and the robot 
direction changes or between the video system yaw and the kinaesthetic way it is operated: P. Peruch, at the end of his 
presentation of [Per.94], put forward an additional commentary about operators sickness provided v/hen a l-to-2 ratio 
exists between a head yaw controlling the direction of a virtual robot and the robot gaze direction, in order to give a 
better anticipation of a curve. This disorientation, obviously due to a sensory conflict, is our concern when using a 
periscope camera and must induce the camera command choice. 

SOLUTIONS TO AVOID VISION SICKNESS 

We propose different solutions (the list is non-exhaustive) to avoid each type of vision sickness in our telepresence 
systems. Our study takes a range of grounds and gaits into account: the best solution depends on the telelocomotion 
nature. Solutions are gathered according to the DOF on which they act. 
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Tilt Sickness Avoidance 

The tilt £ due to the pitch phenomenon of the platform (the "back” of the robot) is represented in Figure 5. In order to 
suppress pitch sickness, it is possible to intervene at least at three stages: 


• Obviously, to avoid the pitch of the robot when it walks it is possible to keep a and s constant. This tilt can be 
minimised (at least completely compensated) as described in [Khe.95], by reducing the attainable domain of the 
legs’ ends given an intermediary platform height, with controlled pneumatic jacks’ lengths. Therefore, this solution 
is efficient in a flat terrain but has disadvantages on a very unstructured one: the leg height compensation 
unfortunately leaves the variations due to the terrain unchanged. 



Figure 5. Pitch of RAMSES 


• In a similar way a controlled camera platform can compensate the tilt with an identical £, not restraining the leg 
attainable domain. This solution can directly use the results of [Ber.96]: it deals with an anthropoid stereoscopic 
foveated vision system with 4 DOF, 2 for its head motion and 2 for its eyes motion: « The Vestibulo-Ocular Reflex 
couples the movement of the eyes to the motion of the head, thereby allowing an organism to keep its gaze fixed in 
space. This is achieved by causing the motion of the eyes to be equal and opposite to the motion of the head. The 
VOR control system is typically modelled as a transformation from head velocity to eye velocity. » [Jor.90] cited in 
[Ber.96]. This solution however has the same disadvantages as the previous one. 

• A more behavioural solution consists of applying the techniques described in [Ber.95], the additional tilts are due 
to the legs and to the terrain being taken into account. An optic flow technique using only a few points in the real or 
virtual image can be associated with a gyroscope placed on the robot platform to avoid lags. 

• On either flat or unstructured terrain, the behavioural solution can be aided by an additional leg variation 
suppression (pitch suppression). 


However, in the behavioural solution, it is necessary to determine the gaze control in both teleoperated and automatic 
modes where it appears that they are non-trivial problems. They are far beyond the scope of this paper but are the 
purpose of intensive research in our telepresence design frame. In teleoperated mode it implicitly deals with the 
interpretation of the operator intention declaration. In automatic mode, the gaze control can not be directly derived from 
previous studies where it is believed that « attention is captured by motion » [Ber.96]. From a camera point of view on a 
mobile robot, the entire environment is moving. 

A relation must be found with the known concept of visual search [Wol.94] cited in [Ber.96]. 
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Terrain Types 

Flat 

Unstructured 

Authorised Speeds -► 


Low 

Low 

Authorised Locomotion Mode 


Teleoperated or 
Automatic 

Teleoperated 

Authorised Teleoperation Mode -► 

Navigation 

Mission or Navigation 

Mission 


Variations compensations 
required 

Legs 

Legs + Ground 

Periscopic 

a) pitch suppression 

sufficient 

not sufficient 

vision system 

b) camera platform 
compensation 

better than pitch suppression 

not sufficient 

(farther 

vision) 

c) horizon tracking 

usable and efficient 

usable and efficient 


Variations compensations 
required 

not relevant 

Legs 

Legs + Ground 

Stereoscopic 

a) pitch suppression 


sufficient 

vision system 

b) camera platform 
compensation 

not relevant 

better than pitch suppression 

(nearer vision) 

c) object tracking 


usable and efficient 


Table 1. Efficiency of the different compensation methods to avoid tilt sickness according to the terrain nature 


The behavioural solution seems to be the best to avoid tilt vision sickness. However, the best solution available depends 
on the nature of the telelocomotion. Table 1 synthesises our research according to the different terrain types and paces. 
The latter matches the several locomotion modes in teleoperation described in 0. 

Roll Sickness Avoidance 

To avoid roll of the visual feedback, it is possible to use the same techniques as those analysed for the tilt. However, it 
must be noted that if a correspondence may be made between the observations in [Gri.87] and vision sickness, it is more 
relevant in the case of roll than tilt. A. Griffin proposes a method to predict motion sickness in marine or other 
environments where vertical oscillation occurs at frequencies below 0.5 Hz. We intend to investigate a parallel 
configuration with horizontal picture oscillations during roll. 

Yaw Sickness Avoidance 

Taking into account the results of [Per.94] already discussed in section 0, it is important to precisely define the vision 
system yaw command: 

• The periscopic camera must be independently operated in yaw from the robot direction in order to allow 
anticipation and localisation in the robot world. Depending on the application context, we propose the use of either a 
high-definition vision helmet or a head tracking system with screen display. However, we insist on the avoidance of 
ratios other than 1-to-l between the head motion and the image display movement to suppress the operator 
disorientation and yaw sickness. Regardless, it must be added that during robot halts, a manual command must 
permit a horizontal investigation of the world on 360°. 

• The stereoscopic video system must be operated by the same interface as the periscopic system, mostly during 
halts, without forgetting the disconnectable automatic facilities. 
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Generally, we must retain, too, the solution presented in [Oma.93] to avoid what we called vision sickness: a prediction 
to smooth the feedback image display derived from the Observer Theory and Kalman filters. 

EXPERIMENTS 

Experiments are in progress to study the relevance in Telelocomotion of a Behavioural Transform by means of gestures 
produced by a dedicated optical fibre glove to operate a legged robot in comparison with the use of a joystick. Variance 
analysis is used to interpret the results. A latin square in experiment protocols provide the order in which the groups of 
subjects are experiencing the different situations based on different vision feedbacks on a unstructured terrain. Time to 
perform the tasks and collisions with the real environment are measured. 

As human interaction with a machine is often oriented towards application, the teleoperation of a mobile robot under 
geometrical constraints is not easy to achieve if no training period occurred previously for the operator. A footbridge 
exists between training and relevant missions or navigation tasks: a real activity with a mobile robot can be prepared in a 
micro-world - a virtual world or an Augmented Reality world. Pre-tests classify subjects in groups according to their 
skill level in driving a point on a screen with a joystick. 
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ABSTRACT 

Canadian Forces (CF) pilots and landing safety officers require intensive training to develop the individual and team 
skills required for safe helicopter deck landings. These skills are currently acquired at sea, following individual training 
with independent simulators unequipped with visual displays. 

DCIEM is exploring the feasibility of using commercial, off-the-shelf technologies as the essential components for 
simulators for training the pilot of the Sea King helicopter and the landing safety officer (LSO) of a Canadian Patrol 
Frigate (CPF). The objective of this project is to assess virtual reality and computer networking technologies that could 
be exploited in the development of a federation of interconnected, low-cost simulators. The human factors of visual and 
motion cueing, and coupling of the simulators, present the major technical challenges to the project s success. 

This paper will describe the exploratory development models, some preliminary reactions, and the experimental plan 
proposed to assess the training effectiveness of the helicopter simulators. 

BACKGROUND 

In 1993, DCIEM completed the exploratory development of the Maritime Surface/Subsurface Virtual Reality Simulator 
(MARS VRS) that made use of virtual reality technologies for training ship handling skills. A low-resolution, biocular, 
head-mounted display was employed to provide a student officer-of-the-watch (OOW) with a view of his bridge and the 
surrounding environment, including other ships, so that he could acquire the skills needed to perform formation 
manoeuvres. Visual judgements of the ship’s relative position, orientation and their rate of change guide the OOW’s 
behaviour as do auditory communications among the bridge crew. Voice recognition was used to interface the OOW with 
a surrogate bridge team that responded to his spoken commands and automated voice production was used to provide 
orders to the student as a yeoman would normally read them aloud; different voices were used to provide the verbal 
information that other members of the bridge crew would normally speak aloud during the conduct of a manoeuvre. The 
behaviours of the other ships in the formation were guided by textbook solutions particular to the sequence of orders that 
formed the lesson plan. 

The results of a transfer-of-training experiment, which compared the performance of students trained with the simulator 
for a week to the performance of students who followed the regular programme of instruction that made use of a gate 
vessel, indicated that the MARS VRS simulator better prepared students for formation manoeuvres when they had to 
perform them for the first time at sea with a mine sweeper. Simulator sickness questionnaires indicated incidence rates 
and severity reports similar to those of conventional aircraft simulators and tests of postural stability using a force plate 
found no evidence of ataxia associated with training. 

On the basis of these positive outcomes, three MARS VRS simulators were later interconnected. Each was operated by a 
qualified officer, at widely separated locations. Although the simulators were physically apart by as many as 5000 
kilometres, distributed interactive simulation permitted the officers to conduct formation manoeuvres within the virtual 
environment as proficiently as they would be expected to perform them at sea, where they are normally within 300 
metres of each other. 

HUMAN FACTORS ISSUES 

The results of the transfer of training study indicate that positive transfer-of-training can be obtained with simulators 
using low-resolution, head-mounted displays for training visually based tasks that do not seem to demand high quality 


Paper presented at the RTO HFM Workshop on <( The Capability of Virtual Reality to Meet Military 
Requirements ”, held in Orlando, USA, 5-9 December 1997, and published in RTO MP-54. 
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imaging; ships in formation are large, slow-moving and relatively close, although simultaneously beyond the range for 
human stereopsis. The results of the networking trial indicate that close coupling of the simulations is not a significant 
concern for formation manoeuvres since the ships move slowly and the tolerances for formation manoeuvres are large 
(within 10 metres) relative to the spatial congruency afforded by distributed interactive simulation technologies (i.e., less 
than one metre). 

In comparison, the deck landing tasks present greater challenges to the successful exploitation of virtual reality and 
interactive networking for training. The pilot of the helicopter must make accurate judgements of surface orientation and 
distance well within the range of stereoscopic perception. In the North Atlantic, fog will often obscure the horizon and 
close proximity to the hangar face will limit the pilot’s ability to make visual judgements of orientation or position 
independent of his immediate surroundings. The relative motions between ship and helicopter will therefore present an 
ambiguous visual environment to the pilot, who will not likely be able on the basis of visual cues alone to determine 
whether the ship, for example, rolled to the left with a wave or whether the helicopter rolled to the right with air 
turbulence. Without physical motion cues, the landing safety officer could similarly be confronted with an ambiguous 
situation when he must assist the pilot in achieving an accurate touchdown. Centimetres matter, and close-coupling of the 
LSO’s and pilot’s visual environments, in time and space, seems to be a clear necessity for simulator team training. The 
purpose of the current efforts are to determine whether virtual reality technologies can provide the visual and motion 
cuing needed to perform helicopter deck landing team training, and whether the High Level Architecture, is suitable for 
real-time, interactive networking in this scenario. 

EXPLORATORY DEVELOPMENT MODELS 

In order to assess the efficacy of the available technologies three exploratory development models are being constructed. 
Two will simulate the Sea King helicopter; each will be linked eventually to an LSO simulator and an existing MARS 
VRS. The two Sea King simulators differ dramatically in the cost and complexity of their components. The first, now 
ready for test flights, has been developed by the University of Toronto Institute for Aerospace Studies, employing state- 
of-the-art (SOA) components. The second, now being integrated by DCIEM, employs commercial off-the-shelf (COTS) 
components. A comparison of system components is shown in Figure 1. The LSO simulator, still under development, 
also employs the COTS technologies identified in Figure 1. 


Figure 1 - Comparison of System Components 
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EXPERIMENTAL PLANS 

The experimental plan requires two sets of comparisons, relying on three groups of subjects. The first set of comparisons 
is a reverse transfer-of-training assessment. Once test pilots complete acceptance of the SOA simulator, a group of 
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qualified Sea King pilots will be invited to fly the simulator. They will make many deck landings as students would 
practice them at sea. The experimental conditions will not simulate the recovery systems or provide guidance from 
others. The task will be performed at the limits of the operational envelop for free-deck landings (i.e., at the maximum 
values for the combination of wind and ship motion permissible without use of the tether). A second group of helicopter 
pilots without deck-landing experience will be invited to learn this task by performing it in the simulator. Later, the 
performances of the two groups will be compared. If the SO A simulator provides an appropriate environment in which 
the skilled behaviour of the Sea King pilots can be exercised then they should show little learning and do well. The other 
group of pilots should not do as well initially, and should improve with practice. 

The second set of comparisons is an intersimulator transfer study. The behaviour of a third group of helicopter pilots 
without previous deck landing experience at sea will be compared to the first two after they have been given training on 
the COTS device. If the COTS device provides an appropriate environment for learning deck landing skills, then this 
group of pilots ought to demonstrate positive transfer of training to the SOA simulator if it is validated by the reverse 
transfer-of-training comparison. 

The suitability of the High Level Architecture for team training will be investigated after the transfer studies are 
complete. 

PRELIMINARY INDICATIONS 

An early version of the exploratory development model of the LSO simulator was recently used as a virtual prototype to 
assist naval architects in their effort to redesign the workstation that now limits the LSO’s ability to view the helicopter 
when it makes a starboard approach to the deck of a Canadian Patrol Frigate. A VR4 head-mounted, binocular display 
made by Virtual Research was used to display a virtual environment generated by an Silicon Graphics Infinite Reality 
Engine. The computer generated imagery consisted of the howdah, Sea King, hangar and deck of the CPF. The visual 
representation of the howdah was matched to different plywood mockups that constrained the movements of the 
observers realistically. LSOs viewed animations of the Sea King as it approached and landed upon the deck. They were 
very positive about the potential use of the visual imagery for the deck landing tasks. This is an encouraging result since 
LSOs are also deck qualified pitots. 
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ABSTRACT 

Historically NASA has trained teams of astronauts by bringing them to the Johnson Space Center in Houston to 
undergo generic training, followed by mission-specific training. This latter training begins after a crew has been 
selected for a mission (perhaps two years before the launch of that mission). 

While some Space Shuttle flights have included an astronaut from a foreign country, the International Space Station 
will be consistently crewed by teams comprised of astronauts from two or more of the partner nations. The cost of 
training these international teams continues to grow in both monetary and personal terms. Thus, NASA has been 
seeking alternative training approaches for the International Space Station program. 

Since 1994 we have been developing, testing, and refining shared virtual environments for astronaut team training, 
including the use of virtual environments for use while in or in transit to the task location. In parallel with this effort, 
we have also been preparing applications for training teams of military personnel engaged in peacekeeping missions. 
This paper will describe the applications developed to date, some of the technological challenges that have been 
overcome in their development, and the research performed to guide the development and to measure the efficacy of 
these shared environments as training tools. 

INTRODUCTION 

Many Space Shuttle flights include an astronaut from a foreign country. These astronauts usually carry out most of 
their training at the Johnson Space Center in Houston or in special facilities at other NASA centers. These astronauts 
often relocate their entire family to Houston. Thus, the support of international crews for the Space Shuttle has been at 
great cost to the nation from which the astronaut comes and has had a high personal cost for both the astronaut and his 
or her family. The International Space Station (ISS) program is a partnership between NASA, the European Space 
Agency, the Japanese equivalent (NASDA), the Canadian Space Agency, and the Russian Space Agency. The four- 
person crews that will staff the ISS will be derived from the partner nations and will also include guest astronauts from 
other countries. For years those responsible for training astronauts for the ISS have grappled with the issue of training 
these international crews. In order to reduce the costs of such training and to address the “political” issues of where 
training should occur, the NASA training community has sought means of training teams of astronauts, to some extent, 
while each astronaut remains in his or her home country. 

Since 1994, we have been exploring the application of shared virtual environments for the training of international 
teams of astronauts. This task has encompassed a number of issues. Among these are necessary communications 
bandwidth, communications latencies, data transfer protocols, interaction metaphors, human figure representations, 
necessary fidelity (visual, audio, haptic), navigation methods, training transfer, and the psychology of team building. 

In addition, the long duration and complexity of ISS missions precludes training personnel for all possible operations 
prior to the mission’s commencement. Moreover, certain skills and knowledge acquired prior to a mission may 
degrade during the month’s-long duration of typical ISS stays. Thus, VEs are being investigated to serve as delivery 
mechanisms for “just-in-time” training. In this case these systems could provide training just prior to the performance 
of a critical task for which no training was delivered prior to the flight or for which skills or knowledge may have 
degraded. It should also be possible to transmit new or updated databases to such space-based VEs to address specific 
unforeseen problems. 

HUBBLE SPACE TELESCOPE MISSION 

In December, 1993 the Space Shuttle captured the Hubble Space Telescope (HST), and astronauts carried out a 
complex set of procedures to correct its mirror’s optical problems, replace and upgrade certain instruments, and carry 
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out planned maintenance (see Figure 1). This mission, by 
most measures, was the most challenging mission ever 
performed in the history of the United States’ manned space 
program. Preparation for the mission was the most careful 
and detailed and brought unique resources to bear for the 
training of both astronauts and the ground-based flight team. 
In order to achieve the maximum state of readiness, virtual 
environments were used, for the first time, in the training of 
the flight team [Loftin, 1995]. This use of virtual 
environments for training joins a small set of applications 
that demonstrate the efficacy of this technology for training. 
Shortly after the conclusion of this mission, interest in using 
shared virtual environments to train international teams of 
astronauts emerged as a high priority within NASA’s 
training community. NASA, the University of Houston, and 
the Fraunhofer Institute for Computer Graphics joined 
together to pursue an initial effort to determine the 
feasibility of a shared virtual environment for team training. 
The context for the feasibility study took the form of a 
simple extravehicular (EVA) simulation that engaged two 
astronauts, one located at JSC and the other in Darmstadt, 
Germany to reenact a portion of the 1993 HST mission. In 
this simulation, the two astronauts changed out the Solar 
Array Drive Electronics (SADE) in the HST. 

Months of preparation culminated in a demonstration on 
September 20, 1995, as Astronaut Bernard Harris 
(physically located in Houston at the Johnson Space Center) entered a virtual environment with Astronaut Ulf Merbold 
(physically located at the Fraunhofer Institute for Computer Graphics in Darmstadt, Germany). Their shared 
environment consisted of models of the Space Shuttle payload bay and the Hubble Space Telescope. The two 
astronauts spent over thirty minutes performing the major activities associated with the changeout of SADE. Their 
work included the real-time hand-off of the replacement SADE in exchange for the original SADE. At the conclusion 
of the task the two astronauts shook hands and waved good-bye [Loftin, 1997]. Interviews with the astronauts after 
this experiment revealed their support of and interest in this mode of training. They confirmed its utility for mission 
planning and the familiarization phase of training and acknowledged its value in reducing travel and personal 
segregation from their families and home environments. The results of this singular experiment have led to sustained 
funding from NASA to continue developing the technologies needed for this type of training and to design and 
conduct experiments to identify human-human and human-machine issues that require solution if this approach to 
training is to become part of the baseline training for the International Space Station. 

INTERNATIONAL SPACE STATION 

Based on the success of the Houston-Germany experiment described in the previous section, we have been developing 
a testbed around the International Space Station. The training focus of this testbed is the conduct of scientific 
experiments and the execution of repair and maintenance operations within the 1SS. Models of the interiors of selected 
ISS modules have been created (see Figure 2) and populated with both highly detailed, interactive models (polygonal) 
of the elements for which training has been developed along with detailed but non-interactive models (texture mapped) 
of the remaining interior elements. The racks of equipment for which training is delivered in this virtual environment 
comprise the Biotechnology Facility (BTF). Prototypes of these racks are currently on-board the Russian Mir Space 
Station, affording opportunities to investigate training transfer for currently-performed tasks (see Figure 3). 

A major distinction between this experimental testbed and the environment used in the Houston-Germany experiment 
described earlier is the transfer of the training context from extravehicular activities (EVA) to intravehicular activities 
(IVA). Whereas in the EVA environment astronauts were represented as “suited” individuals, employing rather 
simple graphics to represent their bodies, in the IVA environment, high fidelity human models were developed to 
serve as avatars for the actual participants in virtual environment training. This approach sets a much higher standard 
for the fidelity with which the humans are represented and challenges the developers to find a reasonable compromise 
between fidelity and graphics display performance (see Figure 4). Training participants are instrumented with sensors 



Figure 1. A team of astronauts prepares to 
maintain the Hubble Space Telescope during the 
December, 1993 mission 
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Figure 2. Cross-sectional view of ISS science module 



that enable real-time tracking of critical portions of their bodies (typically the head, torso, and wrists). Using fast 
inverse kinematics [Tolani, 1996], we have implemented avatar motion for the arms that reasonably represents the 
actual motion of the human arm during the operations carried out by the tracked real arm. Work is underway to 
further refine these inverse kinematics and achieve 
even faster performance and more natural positioning of 
the entire arm [Yang, 1997]. This work seeks to 
combine the best features of the analytic approach 
described by Tolani and Badler [Tolani, 1996] with 
experimental results from neurophysiological studies 
[Lacquaniti, 1982; Soechting, 1989a; Soechting, 

1989b]. Leg motion is not implemented in the current 
environment since it does not play a major role (at least 
in the sense of walking on the Earth’s surface) for 
location in microgravity. Eventually, we will have to 
tackle this problem since leg position is important in 
some tasks where the human must exert substantial 
forces or torques on objects. 


Figure 3. Model of the Biotechnology Facility in 
the ISS and on the Russian Mir Space Station 


The current testbed has been implemented at three sites 
to support experiments in training two astronauts while 
other personnel (trainers or managers) are able to view 
the actions of the trainees via a “stealth” mode. That is, 
the stealth view is a stereo camera view that can be positioned anywhere in the virtual environment but has no 
manifestation in that environment that can be perceived by the trainees. Final experimental procedures are now being 
developed in preparation for controlled experiments using this testbed. The goal of these experiments will be a 
training transfer study to the “real” hardware (either in the Russian Mir Space Station or on the ground once the 
equipment is returned to Houston). 



Figure 4. Avatars are used to represent astronauts in 
the ISS virtual environment 
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It should be noted that a major current thrust is directed at enabling the use of virtual environments in space to support 
“just-in-time” training. Such training, in low Earth orbit, might include environments shared by astronauts in the ISS 
and ground-based engineers and other professionals. Such an approach has been suggested for both emergency repair 
of complex systems and the conduct of medical procedures in response to serious injuries or illnesses that might be 
encountered during long-duration space missions. 

PEACEKEEPING OPERATIONS 

With support from the Office of Naval Research, a group (comprised of the University of Houston, the University of 
Pennsylvania, George Mason University, LinCom Corporation, and Lockheed-Martin) has been conducting basic 
research and applying its results to develop a prototype application to address training of military personnel in 
Peacekeeping Operations. Work has been carried out in two areas of basic research: team training in virtual 
environments and the utility of virtual environments to support nonverbal human-human communication. 

The context for the virtual environment application that is under development is a checkpoint in a Bosnian setting (see 
Figure 5). This scenario offers a versatile testbed for investigating the use of shared virtual environments for training 
small units (fire teams and squads) for carrying out peacekeeping operations in a potentially hostile location where the 
focus is on employing the Rules of Engagement in the face of potentially unfriendly civilians and an enemy operating 
in a covert manner. 



Figure 5 - A U.S. soldier manning a Bosnian 

This application is being used to investigate the degree of training transfer to similar, real-world settings (see Figure 
6). In particular, we seek to challenge the trainees with distracters and to enable them to experience the consequences 
of failure to employ correct procedures and follow the Rules of Engagement. A typical training scenario might 
employ a distracter (an individual running across the soldier’s field of view) aimed at drawing the attention of the 
covering soldier away from his teammate while that partner is engaged in checking the identification of the driver of a 
car stopped at a checkpoint. If the trainee’s attention is drawn from “covering” his teammate, the consequence could 
be the driver taking a weapon from concealment and shooting the soldier who is attempting to check his identification. 
Thus, the results of distraction from one’s assigned mission, can be forcefully brought home to the trainee. 

In tandem with this application development we have been conducting a series of experiments in both team training 
transfer [Bliss, 1997] and nonverbal communication. The first experiments have focused on team navigation skills. A 
larger study is now underway, in collaboration with the Army Research Institute (Orlando Field Unit), to determine 
how well navigation skills acquired as a team in a virtual environment transfer to the setting represented by the virtual 
environment. 
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Figure 6. Scene from Peacekeeping Operations 
Environment 


The second set of studies has not yet been subjected to peer review, but results of two separate experiments have been 
analyzed. These results demonstrate that subjects in typical virtual environments can reliably recognize stereotyped 
and some subtle human facial expressions. The reliability of recognition is comparable to that achieved by the 
subjects viewing photographs depicting the same expressions. Experiments are now being designed to investigate the 
degree to which the application that we have developed can provide training that is transferable to real-world 
analogues. 

IN SITU TRAINING 

NASA astronauts have consistently noted the need for training during spaceflight, even relatively short-duration Space 
Shuttle missions. Apparently they believe that some skills, such as landing the Space Shuttle, deteriorate during a two- 
to-three-week-long mission. In response, NASA has developed a PC-based system, delivered via a laptop 
computer, for landing refresher training. As mission lengths grow (e.g.. International Space Station stays and 
future Lunar/Mars missions) from weeks to months and years, NASA must systematically address in situ training for 
crews. Clearly, much of this training can be provided via standard computer-based techniques. However, there are a 
significant number of tasks that demand interactive, three-dimensional representations in order to achieve the needed 
degree of realism (both visually and through body orientation and manipulation by hand). For example, astronauts 
currently plan their body placements for many tasks using virtual environments. Thus, research and development is 
underway to address the delivery of some training in space using virtual environment technology. The first virtual 
environment system will be flown aboard the Space Shuttle in mid-1998 as a part of Neurolab (STS-90). Although 
this system is intended for use in a human perception experiment, its development and flight clears the way for future 
space-based experiments that utilize virtual environments for training. It is anticipated that such an experiment will be 
conducted during a mission in late 1999 or early 2000. The success of this type of experiment will likely lead to the 
incorporation of a virtual environment within the International Space Station. 

In addition to applications during long-duration Space Shuttle missions and aboard the International Space Station 
(both of which occupy low earth orbits), it is probable that virtual environments will be required for any human 
mission to the lunar surface or to Mars. In the latter case, transit times will exceed one year with total mission time 
occupying three or more years. Moreover, communication lags can reach thirty minutes and there can be significant 
periods (months in length) with no communication at all due to particular Earth-Sun-Mars alignments. 

IMPLICATIONS FOR NATO 

Other papers in this workshop have addressed the use of virtual environments for a variety of military training 
applications. The specific projects described in this paper, while not designed for typical warfighting training, do 
provide a basis for developing and/or improving virtual environment training for a variety of modern military 
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missions. In particular, the application of virtual environments for peacekeeping mission training should be of interest 
to NATO, given its current role in Bosnia. 

The use of virtual environments (both individual and shared) for small unit mission planning and training offers a 
number of benefits. For example, unit commanders can plan and review tactics with appropriate superiors and 
subordinates. Whole units (or parts thereof) can rehearse entire missions or critical mission elements to resolve such 
issues as timing and personnel placement. Perhaps most importantly many options for attaining a specific mission 
objective can be quickly explored during the mission planning phase. Such an approach will likely result in a 
significant increase in the probability of mission success and a concomitant reduction in the likelihood of casualties 
(both military and civilian). 

Since NATO is a multi-national organization joint operations, even at the small unit level, are likely. Shared 
environments offer the ability for units, comprised of personnel from more than one service and/or more than one 
nation, to rehearse a mission while they are still in their “home” locations or even in transit to the site of operations. 
Such an application of virtual environments could reduce response time. In addition, the time spent in these shared 
environments could lead to increased levels of trust and cohesion among individuals who have never met prior to the 
conduct of a mission. Finally, a characteristic of modern military operations is the fluidity of team composition. 
Teams may be constituted for a specific objective within the context of a larger mission. Such teams may only exist 
for hours or days, at most. The achievement of mutual trust and team cohesion may be impossible if traditional team 
training approaches are used. Again, shared virtual environments between participants in their “home” locations or 
while in transit to the site of operations may have give these fluid teams the “edge” needed to succeed in the modern 
battlespace. 
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FRENCH MILITARY APPLICATIONS OF VIRTUAL REALITY 
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49460 MONTREUIL JUIGNE, France 


INTRODUCTION 

France is now applying virtual reality techniques to military purposes. In parallel, France is conducting studies related 
to the psychological effects of immersion. 

Our approach is based on the creation of synthetic environments in which men are more or less immersed. Besides we 
have been searching for the means that will give the best price/quality ratio. This resulted in creating environments 
which are both simplified and improved with respect to the actual environment. This document describes some current 
applications. 

The 5 H’s 

First, it is necessary to define the range in which the virtual reality techniques will be applied. In our opinion, these 
techniques can be applied to five fields, i.e. the 5 Hs: 

1. Help for design, 

2. Help for test, 

3. Help for training, 

4. Help for mission preparation, 

5. Help during operations. 

1. This application is currently being undertaken at the Etablissement Technique d Angers (ETAS). The purpose is 
to be able to simulate vehicle compartments and to analyse them in dynamic conditions. 

2. It will be possible to use the above application to perform tests on currently developed equipment. This 
corresponds to the second type of application. 

3. The third application is for personnel training and instruction. 

4. The fourth application is related to the preparation of missions. Works are in progress to help special units to 
prepare interventions. The purpose is to replace the old sandboxes or even the real-size site reconstruction 
techniques with synthetic environments where soldiers can repeat the various actions to be carried out. 

5. The fifth application for which we will give an example, is related to the use of these synthetic environments as 
an assistance during actual missions. 

To illustrate training assistance, we will give three examples corresponding to different degrees of immersion. The 
first one is for parachuting, the second one for resuscitation gestures, and the third one is a film related to Leclerc 
tanks crew training. 

Finally, an example will illustrate the use of these techniques as an assistance during a real mission. 

Parachuting 

The developed simulator is very similar to the one developed by Systems technology, Inc. The data processing means 
involved in its creation are not expensive. The synthetic environment is a simplified representation of the training 
environment that includes part of the Etablissement Technique dAngers (ETAS). The parachute drift laws have been 
established from data recorded in flight by test parachutists of the Airborne Centre of Toulouse (CAP). 


* Currently at: 7, Rue Roger, 92140 CLAM ART, France 

A special effort has been made to simplify at the maximum the simulation supervision operations. The jump can be 
recorded and then shown to the trainee. It is also possible to modify meteorological parameters. 


Paper presented at the RTO HFM Workshop on “The Capability of Virtual Reality to Meet Military 
Requirements”, held in Orlando, USA, 5-9 December 1997, and published in RTO MP-54. 
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The trainee can see both the parachute and his/her feet. In this application, the parachutist and his/her harness are 
immersed. The only "non-realistic" interface is the virtual reality helmet. The control actions are performed by acting 
on the real controls of the parachute. 

This type of simulator, with its simple architecture and the reduced number of interfaces, has been received positively 
by professionals. 

Front-Line Medical Care 

The basic characteristic of France is advanced front-line medical care with the practice of medical and surgical 
operations on the actual combat spot. 

The current type of military intervention is characterized by the engagement of a low-strength unit for which medical 
support is provided by a unit doctor who has a front-line armoured ambulance which can be used to evacuate wounded 
personnel. Very often, this involves emergency resuscitation which requires the practice of elementary resuscitation 
gestures. These gestures are characterized by the fact that they must be done blind, whether inserting a perfusion in the 
jugular vein, anaesthetizing the brachial plexus or placing a cannula in the trachea. Sometimes these operations have to 
be done inside the vehicle, in combat gear, with a heavy helmet and a bullet-proof vest; therefore, it takes place in a 
narrow, cramped space which is often hot and noisy and sometimes the individual is restricted in movement. The 
posture is often uncomfortable with the presence of combat stressors, in particular enemy fire. In addition, these 
operations are carried out by general practitioners. 

In this context, Chief Medical Officer TRIFOT, director of CITERA in Lyons, had the idea of launching a training 
project using virtual reality techniques for the teaching of elementary resuscitation gestures. This project is partly 
financed by "Mission Innovation" at the DGA and involves different partners which, in addition to CITERA, include 
the Human Factors division of Etablissement Technique d’ Angers (ETAS), the University of Angers, the Laboratoire 
Robotique de Paris and a firm of architects (Pierre Granjean). 

At present, the tool is used as follows. The students learn to perform gestures by initially regarding the in-depth 
anatomy of the neck and the route followed by the needle in it. The pertinent elements of this anatomy are reinforced 
by colours. Subsequently, the student repeats these gestures while only seeing the skin but having an enhanced return 
of the manual sensation. Finally, he performs the activity with the same work field as in reality, i.e. with little manual 
sensation and a view of just the surface of the region. In terms of verification when the act is performed, the student 
can see the result by displaying the position of the needle in depth. 

The last stage of training consists in doing this task in the GEPAT (Generateur d’Environnement Physique Aggressif 
pour le Travail). In addition to confinement, this generator, developed under the responsibility of Laurent Todeschini, 
ETAS, produces noise, heat, movement and hypoxy. The task is performed wearing combat gear, a bullet-proof vest 
and a heavy helmet. 

To conclude, such a product should enable doctors to be trained how to operate in any critical situation through 
learning not only the technique but also the ability to face up to it. 

The studies already carried out in the framework of this project lead us to draw a parallel between this front-line 
medical facility and manual mine-clearing actions: this resulted in the study of a device for exploration of 
heterogeneous layers, in both the mine-clearing and the medical fields. 

Leclerc Tank Crew Training 

The film shows a specific application where the immersion is obtained via a tank simulator that allows interactive 
training of the three crew members. This device is now operational and used to train all the Leclerc tank crews. 

Assistance during Missions : Definition of a Virtual Area for Action (DEVA - Definition d'un Espace Virtuel pour 
i'Action) 

The last presentation of this summary deals with the use of synthetic environments as an assistance to drive vehicles. 
The developed device includes an ETAS geographical database used to supply the drivers with a full-size reproduction 
of the field with a precision of +/- 2 cm for roads and 1 to 10 m for the rest of the area. In addition to this geographical 
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data, it is possible to integrate information such as road signs or identification of dangerous areas. As the driven 
vehicle is equipped with video cameras, it is possible to supply the pilot with a navigation aid by superimposing on the 
real images the synthetic information generated by the application. 

The second driving mode is the "blind" mode (i.e., without seeing the outside of the vehicle and helped only by the 
synthetic geography). Note that the position of the vehicle is given with a precision lower than 7 cm by a GPS coupled 
with an on-board system. The tests performed in these conditions show that the pilots could drive their vehicles in 
"blind" conditions at a speed of approximately 45 km/h. 

In addition to the above driving modes ensured by this device is the "God's eye view" mode: it can be compared with 
the way a child drives a remotely-controlled car. The future trajectories of the vehicle can be anticipated since this 
mode offers a long-distance vision. We can easily imagine its tactical applications. It is also possible to superimpose 
on this synthetic image the image of the actual landscape, for example from an infra-red camera showing obstacles. 
This allows driving by night without lights and in foggy conditions. 

This application offers a high potential and in the future, instead of these screens, it should be possible to make a 
projection of the synthetic environments directly on the windscreen. 
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FORMATIVE EVALUATIONS OF THE 
VESUB TECHNOLOGY DEMONSTRATION SYSTEM 

Robert T. Hays, Ph.D. 

Naval Air Warfare Center Training Systems Division 
Code 4962 
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Orlando, FL 32826, USA 


BACKGROUND 

Decreasing military budgets, reduced training infrastructure, and increasing complexity of weapon systems and 
missions mandate the exploitation of innovative advanced training technologies. In recent years, training developers 
have recognized the potential of virtual reality (VR), often called virtual environments (VE), as a flexible and effective 
training medium. A prime candidate area for examining the effectiveness and usability of VR systems is the training 
of the submarine surfaced ship handling task. Although land-based simulator facilities currently exist for training 
Submarine Piloting and Navigation teams, these systems do not provide detailed harbor and channel ship handling training 
for the Officer of the Deck (OOD). OOD training is primarily obtained from on-the-job experience, which is adversely 
impacted by the operational constraints of the Submarine Force, and the limited surfaced steaming time of submarines. 
Training that will expose junior officers to a variety of geographical and environmental conditions is very limited since 
most Commanding Officers place their most experienced OODs on watch during these challenging evolutions. Therefore, 
an alternative, high-fidelity, simulation-based training capability is needed. A VR-based simulation may provide this 
necessary capability if it is both effective and user-friendly. 

OBJECTIVE OF VESUB 

The goal of the Virtual Environment for Submarine OOD Ship Handling (VESUB) project is threefold: (1) to develop, 
demonstrate, and evaluate the training potential of a stand-alone virtual reality-based system for OOD training; (2) to 
integrate this system with existing Submarine Piloting and Navigation (SPAN) training simulators; and (3) to determine the 
viability of VR technology as a training tool. 

DESCRIPTION OF VESUB 

The VESUB technology demonstration system uses a high-resolution head mounted display (HMD) to provide the trainee 
with a simulated 360 degree representation of the visual environment containing many of the required cues associated with 
harbor and channel navigation as well as accurate cultural features and varying environmental conditions. Voice 
recognition and synthesis are used to provide communications training. Visual scene rendering, computation of harbor 
currents, wind effects, hydrodynamics of own ship and traffic ships require state-of-the-art hardware and software, 
including: 


• Silicon Graphics Onyx Image Generation Computer (Infinite Reality configuration) 

• n-Vision Datavisor HiRes HMD (1280 x 1024 pixels resolution) 

• Polhemus 3 space Fastrack Head Tracker (Magnetic) 

• ModelGen2 from Multigen, Inc. (creation of models and terrain) 

• Vega Marine from Paradigm Simulation, Inc. (marine visual effects) 

• HARK, Speaker-Independent Voice Recognition software 

Additional details on VESUB hardware and software may be found in Hays, Seamon, and Bradley (1997). 

VESUB DEVELOPMENT PHASES 

The VESUB project was organized in three phases: 1) requirements determination; 2) formative evaluations; and 3) 
training effectiveness evaluations. 


Paper presented at the RTO HFM Workshop on u The Capability of Virtual Reality to Meet Military 
Requirements”, held in Orlando, USA, 5-9 December 1997, and published in RTO MP-54. 
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Requirements Determination Phase 

The requirements determination phase was the first year of the project. During this phase, a simplified feasibility 
demonstration system, developed under the NAWCTSD exploratory research Virtual Environment Training 
Technology project, was used to elicit VESUB system requirements from Navy subject matter experts (SMEs). The 
initial functional requirements were documented in a NAWCTSD special report (Tenney, Briscoe, Pew, Bradley, 
Seamon, & Hays, 1996) and used to direct the contractor during the development of the VESUB technology 
demonstration system. 

Additional VESUB functional requirements and training objectives were developed around the organizing concept 
“Seaman’s Eye,” defined by the VESUB research team as follows: 

Seaman ’_s Eye : The total situation awareness of the ship handling environment and the ability to 
safely maneuver the vessel in all conditions . 

The perceptual and cognitive components of “Seaman’s Eye” were identified and used to help determine hardware, 
software, and instructional requirements. Details on how the perceptual components were used to help select VESUB 
hardware may be found in Hays, Castillo, Bradley, and Seaman (1997). 

Formative Evaluation Phase 

Formative evaluations serve as initial tests of the configuration and instructional methods included in the training system. 
These evaluations help the training developer to avoid or correct costly mistakes before they are finalized in the 
configuration of the training system. The software development and hardware integration for VESUB was accomplished 
under contract by Nichols/Advanced Marine. The contractor developed the VESUB system at their offices near 
Washington, DC. The VESUB formative evaluations were conducted on a duplicate VESUB system in the laboratory at 
NAWCTSD during the second and third years of the project. Each time the development contractor produced an improved 
iteration of the software, it was installed in the VESUB laboratory and evaluated against the functional requirements. Over 
ten iterations of the system have been produced and evaluated. 

Data for the formative evaluations were collected from a wide range of SMEs. These included: full time retired Navy 
submariners; active duty Navy submarine SMEs from the fleet and training facilities; and Navy reservists. 
Recommendation for system improvements were provided to the software developer for improvement of the next system 
iteration. A brief summary of the results of the formative evaluations are presented below. 

Training Effectiveness Evaluations 

The training effectiveness evaluation (TEE) phase will include TEEs of the VESUB technology demonstration system. 
The first TEE will be conducted at the Submarine Training Facility in Norfolk, Virginia during January 1998 and the 
second will be conducted at the Naval Submarine School in Groton, Connecticut during March 1998. The TEEs will use 
actual Navy trainees with various levels of experience (novice to expert) to determine the effectiveness of the VESUB 
system and also to help determine how the technology can be integrated into Navy training. During the Norfolk TEE, 
VESUB will be connected to the SPAN trainer to determine if the modern VESUB hardware and software can 
communicate with the older SPAN system. If feasible, the future operational VESUB systems will be interfaced to SPAN 
trainers to provide interactive OOD training with other members of the Navigation Team. The results of the TEEs and 
other lessons learned during the VESUB project will be documented for use in the acquisition of the operational VESUB 
systems. 

Acquisition of Operational VESUB Systems 

Current plans call for the acquisition of operational VESUB systems, beginning in 1999. The systems will be installed at 
the Navy’s five major submarine training facilities. The VESUB technology demonstrations system will serve as a 
functional baseline, which will have to be matched or improved. The operational systems will include the functionality of 
the demonstration system and improve on this functionality, based on the results of the TEEs. 
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RESULTS OF THE FORMATIVE EVALUATIONS 

VESUB formative evaluations focused on both the functionality of the trainee interface (e.g., the fidelity of objects in the 
visual scene or the responsiveness of the voice recognition system), and the usability of the Instructor/Operator Station 
(IOS). The results of the IOS usability analysis are presented in Hays, Seamon, and Bradley (1997). This report 
documents twenty-two areas where the design of the operational VESUB system IOS can be improved. 

Hundreds of SME recommendations for improvement of the trainee interface were collected during the formative 
evaluations. These recommendations ranged from very general (e.g., improvements in the representation of equipment 
on the submarine bridge) to very specific (e.g., the logic supporting the “angle on the bow” display on the IOS). The 
recommendations were provided to the system developer and prioritized for inclusion in subsequent software 
iterations. Only a limited number of the recommendations could be included in the demonstration system due to 
budget, schedule, and technology constraints. For example, in March and April of 1997, seventy-three major system 
requirements were reduced to twelve. Two examples of requirements that were not implemented are: time varying 
fog (e.g., fog bank “rolling” in rather than instant reduction of visibility) and secondary propulsion motors (used when 
the rudder or main engine fails and for docking maneuvers). Two examples of requirements that were included in the 
system are: environment sounds (e.g., ship whistles, bells on buoys, wind, and rain); and latitude/longitude 
organization of the databases (rather than an x-y coordinate system as in previous ship handling trainers). An example 
of a partially developed function, which will require improvement in the operational system, is collision detection. 
Finally, several unanticipated requirements were recognized only after initial versions of the software were evaluated. 
An example of these unanticipated requirements is large area database management (required because of the need to 
display both close and distant objects in the visual field). Details on the trade-offs made during system development 
will be documented in a final “lessons learned” report. 

CONCLUSIONS 

The development of a complex training system, like VESUB, requires the dedicated efforts of a team of experts including, 
computer scientists, visual engineers, task SMEs, and instructional psychologists. It is essential that a substantial period of 
formative evaluations be included in the development of any such training system. It is not possible to anticipate even a 
fraction of the problems that will be encountered during system development. Formative evaluations provide the 
opportunity to correct problems before they are fully implemented in the system and become too costly to correct. As 
training systems, VE-based or otherwise, become more advanced, they also become more complex. This complexity 
brings unanticipated difficulties, which will require innovative and timely solutions. Formative evaluations provide the 
opportunity for system developers to apply these solutions at a stage where they can have a positive benefit to training 
system effectiveness. 
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ABSTRACT 

This paper reports on two cases in which Head Mounted Display (HMD) based Virtual Environments (VE) are applied 
to military training. The first case deals with Forward Air Controller training, while the second case is aimed at Stinger 
training. Both applications are subjects of study within the VE research program of the TNO Physics and Electronics 
Laboratory and the TNO Human Factors Research Institute. 

For the Forward Air Controller (FAC) training application a feasibility study was recently performed. Based upon a 
task and training analysis, a prototype FAC training simulator was developed and evaluated. Evaluation results have 
encouraged both the Dutch armed forces and other NATO countries (United Kingdom, Belgium) to seriously consider 
the HMD based FAC simulator as a useful and effective training tool. The paper describes the simulator prototype and 
evaluation results, including the human factors issues that were addressed in this study. 

The Stinger training application is still in an early stage of development. Instigated by a demand of the Royal 
Netherlands Air Force, the research is focused on finding a suitable concept for a mobile training device. The paper 
describes a planned project (to start in 1998) that includes the development of an experimental system in order to 
evaluate several design alternatives. The evaluation has as a goal to determine how human performance is affected by 
the type of display system that is used, the overall system latency and the tracking accuracy. 

INTRODUCTION 

Beginning with the emergence of the technology, the TNO Physics and Electronics Laboratory has been researching 
the application of advanced Virtual Environment (VE) technology to military training and command & control 
problems [1], One of the focal points in this research program is the use of Head Mounted Display (HMD) technology 
to provide a visual environment that fully surrounds the user, the main advantage of HMDs being their much smaller 
size then conventional projection display devices. 

The use of HMD technology in virtual environments for training simulation has always been recognised to be of great 
potential. However, to date TNO has no knowledge of operational military training devices based upon this 
technology. The application of VE technology to the training of Forward Air Controllers and to the training of Stinger 
teams are likely to be among first cases that will change this. 

In the remainder of this paper we provide a description of two research projects that are carried out on the application 
of HMD based virtual environment systems to the Forward Air Controller training problem and the training of Stinger 
teams. 

Forward Air Controller Training Simulator 

Initiated by a demand for more effective training tools at the Netherlands Integrated Air Ground Operations School 
(NIAGOS), the use of VE technology for training Forward Air Controllers was made the subject of a feasibility study. 
In a co-operative effort between the Royal Netherlands Army, the TNO Physics and Electronics Laboratory and the 
TNO Human Factors research institute, a study was carried out to determine whether an HMD based training simulator 
would be a valid and feasible solution to improve training effectiveness. 

The Forward Air Controller (FAC) plays an important role in Close Air Support (CAS) operations. CAS operations are 
performed when air support is requested to attack enemy units that are in close proximity to friendly units. The task for 
the FAC is to guide the CAS pilots in the final stage of their mission such that they engage the correct enemy targets, 
without endangering friendly forces. 

CAS pilots are briefed in advance on their mission objective and all relevant mission data. The brief includes 
information on the target position and describes the position of nearby friendly units, and possible threats the pilot may 


Paper presented at the RTO HFM Workshop on “The Capability of Virtual Reality to Meet Military 
Requirements", held in Orlando, USA, 5-9 December 1997, and published in RTO MP-54. 
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encounter. Based upon the specified target position, the pilot programs his navigation system that essentially guides 
him roughly to the target, taking into consideration the fact that there will be a certain error in the specified target 
position, that the navigation system has a limited accuracy, and that the target may be moving. 

At this point the FAC comes into the picture. Based upon briefing data, the pilot will be able to fly to the target area. 
The FAC then takes over to make sure that the eyes of the pilot are guided onto the correct target and that the pilot is 
aware of the position of friendly units in the target area such that they are not endangered. 

To accomplish his task, the FAC chooses an observation position (OP) in the terrain from where the target area can be 
well observed, while at the same time an unobstructed view is provided in the direction where the plane is expected to 
show up. From the OP, the FAC continuously observes the enemy units, most specifically the designated target, the 
friendly units and (when in sight) the plane. 

The FAC provides guidance cues to the CAS pilot via a UHF radio connection. This radio communication has to 
conform to a NATO standard procedure for FAC operation. 

In situations where it is difficult to find clear marking points in the terrain, the FAC has several ways to create 
artificial reference points: special light reflectors or lamps, smoke grenades, and flares can be used for this purpose. 
Most CAS planes are also equipped to detect laser target designator signals. The FAC then uses a laser target 
designator to point out the target by putting the laser spot upon it. 

Current Training Practice 

A typical initial FAC training course currently takes three to four weeks. The first part of the course, about a week, 
covers theory: air operations in general, CAS operations, NATO standard FAC procedures, map handling, radio 
operation, radio procedures, etc. After this, the practical issues are divided into two parts: low threat and high threat 
operations. 

Low threat FAC operation is commonly considered to be relatively easy. A FAC who is qualified for high threat 
operations, is also assumed to be qualified for low threat. Low threat operation is only trained by using aerial 
photographs as a visualisation tool. The 'pilot' (usually an instructor) is given an aerial photograph of the target area as 
it would be seen from the plane when circling above the area. The FAC trainee has prepared the scenario with a map 
and possibly with observations in the real terrain. The goal of the exercise is then for the trainee to 'talk* the eyes of the 
pilot onto the correct position on the aerial photograph. 

Most training effort is put into high threat operations. Training for high threat FAC operations is currently facilitated 
in three ways in addition to theoretical instruction: 

• Review of video and voice recordings that are taken from a cockpit during FAC operations (so-called HUD 
tapes); 

• Classroom simulation by using a scale model of a target area and a toy plane; 

• Live training runs. 

HUD tapes and scale model simulation are only used in the first week of the course. The remainder of the course is 
filled with live training runs to exercise high threat training as much as possible. Low threat scenarios are usually not 
trained with live training sorties. 

Training Bottlenecks 

Analysis of the current FAC training practice shows a number of bottlenecks that reduce both the effectiveness and the 
efficiency of current training: 

• Moving from classroom training to live training for high threat operation proves to be too big a step, as 
witnessed by the bad results of the first series of live training runs; 

• Limited availability of flying hours for live training sorties often severely impedes the training course (due to, 
for example, cost restrictions, capacity restrictions, flying restrictions, weather restrictions and bird activity 
restrictions.) 
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• Live training is the only effective training tool for high threat FAC operation, but it is a very expensive tool to 
use to this end. 

These bottlenecks have been taken as the starting point for the hypothesis that a new simulation tool will solve them, 
and improve both the effectiveness and efficiency of FAC training. 

Problem Approach 

To test the hypothesis that an HMD-based training simulator would solve the above mentioned bottlenecks, the 
following approach has been followed: 

• A task analysis of FAC operation has been performed; 

• A training objective analysis for initial FAC training has been performed; 

• The training objectives have been mapped on training tools, to find out which training objectives should be 
reached with the simulator; 

• A set of initial user and system requirements has been defined for the FAC simulator; 

• A prototype simulator has been built; 

• an evaluation of the prototype simulator has been performed to test the hypothesis and assess the feasibility of 
the FAC simulator. 

Simulator Solution 

The FAC task analysis and training objective analysis results indicate that FAC training can be improved by providing 
a simulator tool that aims to contribute to the following training objectives: 

• Planning of a FAC operation: 

- Creating a mission brief; 

- Selection of a suitable OP; 

• Pilot guidance: 

- Communication procedure with the pilot; 

- Recognising and adjusting the plane's flight path; 

- Observation of the environment; 

• Battle damage assessment. 

These training objectives are currently covered by either classroom instruction that lacks realism or by live training 
that is not cost-effective. 

The essence of the FAC simulator is that it shall provide a training tool that bridges the gap between the standard 
classroom instruction and the live training. The simulator shall enable the FAC trainees to experience and exercise 
FAC procedures in a realistic way (i.e., with realistic time pressure and realistic visual perception tasks) before they 
are exposed to real planes. 

It is claimed that an HMD-based virtual environment system embedded in a distributed interactive simulation network 
(see Figure 1) will provide this bridge between theory and practice. 


DIS/HLA network 



Figure L The conceptual structure of the FAC training simulator system. 









25-4 


The need for an HMD (see Figure 2) is motivated by the requirement that the FAC must be able to fully observe the 
environment around him, since CAS plane and target are usually in opposite direction. An HMD is advantageous over 
other projection displays in the fact that it provides visual feedback covering the entire 360° azimuth and 180° 
elevation field of regard in a very small and affordable device. Because of this, the HMD is very suitable for tasks that 
rely on spatial perception of an environment, as is the case with FAC operation. 


Evaluation Method 


The simulator solution presented in the previous section was 
prototyped within TNO’s Electronic Battlespace Facility [2]. This 
prototype enabled us to evaluate whether the proposed HMD based 
training simulator solution is a valid and acceptable solution. A 
concise description of the prototype can be found in [3]. For the 
course of this paper, it is relevant to note that: 

• The HMD used for the prototype is an n-Vision HiRes 
stereoscopic display device that displays 1280x1024 pixels in 
each eye. The optics of the HMD project the images onto a 
field of view of 63° horizontally by 34° vertically per eye, 
with a 50% overlap between the two images. This yields a 
resolution of 1.6 arcmin per pixel. 

• The HMD is tracked by an InterSense IS-300 PRO tracking 
device that continuously reports the viewing angles of the 
FAC trainee. The IS-300 uses inertial technology to determine 
the FAC’s viewing direction. 

• The stereo images for the HMD are provided by a Silicon 
Graphics Onyx 2 with a single Infinite Reality pipe. Average 
frame rate was 30 to 60 Hz. 

• Figure 3 depicts a typical database scene as seen by the FAC. 

• While performing simulator exercises, the classroom console 
provides a simultaneous view of the scene as seen by the FAC 
and the pilot as well as a map view of the target area. These 
images are presented on large retro projection screens. 



Figure 2. The FAC simulator uses a 

Head Mounted Display to immerse the FAC 
in the working environment. 


Prototype evaluation has been done in five sessions. In the first two sessions, only FAC instructors were involved, 
whose primary aim it was to assess whether it is possible to simulate FAC operations with the simulator, and secondly 
whether the system is as suitable as an instruction tool. 

After the positive outcome of the first two evaluation sessions, the prototype was to be further evaluated in three 
regular FAC courses as given by NIAGOS. The objective of these sessions was to determine how inexperienced 
trainees would cope with the system and what the learning transfer would be. In the first week of the course, after 
trainees had caught up with theory, simulation training was performed during a single day. Practical constraints limited 
the use of the simulator to only a single day per course - ideally the simulator should be used during the whole course. 

The evaluation is limited in the fact that it has been mainly qualitative in nature. A lot of issues require further 
evaluation and quantification. However, the evaluation that has been performed enables us to answer the main 
question of this feasibility study, namely whether the HMD based FAC training simulator can be a tool to resolve 
current FAC training bottlenecks. 
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Figure 3. A typical view on the virtual target area as seen by the FAC through the HMD 


Evaluation Results 

The main outcome of the evaluation sessions is that the FAC simulator proves to be a valid simulation of FAC 
operation and that it can be used effectively to improve FAC training. 

Improved Training Effectiveness 

The first observation is that the use of the FAC simulator enhances training effectiveness. Taking into account that 
only a single day of simulator training was integrated in the course, it was already observed that the initial live training 
runs after simulator training were more successful than in courses without simulator training. 

The main reason for the fact that the simulator enhances training effectiveness, is that the simulator is a more suitable 
tool to train novice FACs than exercising on live training runs. When using only live sorties as a training tool, the 
trainees will need many runs to get to a certain minimum capability level. Many of these runs will fail. Using the FAC 
simulator for this stage of training demonstrates it to be a more effective way to push the trainees to a desired initial 
capability level. The reason behind this is that simulator training is much more controllable than live training - the 
level of the exercises can be matched with the capabilities of the individual FAC. 


Improved Training Feedback 

The use of a classroom console has proven to be a very valuable instruction tool. The correlation of the FAC’s view 
with the flight simulator view in particular provides trainees good insight into how a pilot perceives the terrain, thus 
learning which cues are best used for visual guidance. 

The use of the scenario freeze and scenario playback feature also demonstrated the value of simulation over live 
training. During live training, there is little opportunity to evaluate runs. Only after the planes have returned to base, 
the HUD tapes can be reviewed and the pilots can give their comments on the runs. Simulation, on the other hand, 
allows direct feedback on the trainee’s behaviour. 
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The improved feedback also facilitates the instructor in the evaluation of a trainee’s performance. It is observed that 
this evaluation is currently done in a very subjective and informal way. The evaluation could be pushed to a more 
objective level by introducing automated support for it. The available DIS/HLA network data and communications 
recording could be a source of information to extract objective scoring data. 

Improved Training Flexibility 

Having the control to fly sorties in the simulated world at any desired time, any desired place, and in any desired 
pattern has proven to be a relief for FAC instructors. Current FAC courses are continuously impeded by uncontrollable 
variables like weather, birds, mechanical problems with planes, etc., which are fully controlled in the synthetic training 
environment. 

Human Performance Issues 

Several important human factors issues have been observed during the evaluation sessions of the prototype: 

• All trainees accept the HMD based simulator as a useful and pleasant tool - no simulator sickness or discomfort 
was reported; 

• To obtain visual aircraft detection ranges that are realistic (5 to 8 kilometres), the aircraft model is scaled up 8 
times (and gradually scaled down to a 1:1 size as it approaches the FAC); 

• Trainees are indeed able to give visual guidance control to the pilot by perceiving the position and orientation 
of the plane; 

• Trainees have some difficulties in getting the right orientation within the terrain (this problem has been reduced 
by displaying a heading reading at the bottom of the screen). 

Technical Issues 

From a technical point of view, the experiments with the FAC simulator prototype have taught us the following 
lessons: 

• The InterSense tracking system is superior over often used electromagnetic tracking systems, considering both 
accuracy and speed - a comparison between the two has shown considerably improved perception capabilities 
for target detection, due to absence of lag and jitter; 

• A large amount of system development effort will have to go into database development - a diversity of high 
quality databases is a prerequisite for a fully operational training simulator. 

• The distributed architecture of the FAC simulator, relying on the standard DIS/HLA concepts, provides ample 
opportunities for the development of a fully functional and extendible simulator. 

Simulator Opportunities 

The current study has only touched upon FAC simulation to a limited extent: the use of simulation for initial training 
of high threat FAC operations. However, simulation brings a lot more opportunities. The full list of applications 
includes: 

• Personnel selection; 

• Initial training; 

• Currency training; 

• Currency testing; 

• Mission rehearsal. 

FACs who already have a combat ready status, will need training to keep up their currency. Currency training with the 
simulator will be a good approach for this purpose. Within NATO, combat ready FAC's have to perform six successful 
runs every half year. A combat ready FAC has once been reported to need eighteen runs to perform six successful 
ones. It would have been much more cost effective if these twelve unsuccessful runs would have been performed in a 
simulator. 
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For currency training, the FAC simulator can be connected to a real flight simulator, possibly situated at a remote air 
base, via the DIS/HLA network. Thus, both FAC and CAS pilot can be trained to keep up currency. 

Once simulation is an accepted tool in FAC currency training and has demonstrated to decrease the number of 
currency test runs that is required to fulfil the NATO currency requirement, simulation may well become a mandatory 
part of currency tests. 

As the application of the simulator is extended to currency training and even currency tests, the requirements for the 
simulator are also extended. More complex scenarios need to be simulated, including threat handling, multiple CAS 
planes and more difficult conditions like night CAS operations. 

For mission rehearsal application, not only the complexity of the scenario is increased, but also the need for a 
geospecific database of the mission area is put forward, as is the simulation of dynamic threats of various kinds 
(enemy aircraft, ground forces, radar, etc.) 

All these opportunities indicate that FAC simulation provides ample means to increase the capabilities of FACs, while 
in the end a strong reduction of cost can be effected due to a reduced number of live training runs that is required. At 
the same time, the live training runs that are still performed - and necessary - are used more effectively. 


FURTHER WORK 

The FAC simulator has been received with great enthusiasm within the FAC community. The fact that both research 
and development on this subject have only been partly touched upon, imply that a lot of work remains to be done. 
Here are some of the topics: 

• Conduct a more quantitative study of the impact of the simulator on training effectiveness and efficiency; 

• Find ways to solve the terrain orientation problem by displaying auxiliary information in the HMD - without 
introducing negative training transfer; 

• Introduce objective trainee evaluation concepts and automated support for this to push the current subjective 
scoring methods to a more objective level; 

• Develop a fully operational FAC simulator. 

• Allow for more complex scenarios, such as threat handling, and night CAS procedures; 

• Extend the concept of the simulator to airborne FAC, heli-FAC, and forward artillery observers (FO). 

CONCLUSION 

The main conclusion of our FAC study has been that introduction of an HMD based simulator for FAC training is 
feasible and will greatly enhance training effectiveness and efficiency. Evaluation of the prototype system within a 
series of three FAC courses has shown that the simulator is accepted by the trainees as a useful tool to learn with and 
by the instructors as a useful tool to teach with. 

By filling the gap between classroom training and live training, the FAC simulator solves all bottlenecks inherent to 
current training practice. The advantages and the enthusiasm of the users that have been observed during the 
evaluation of the prototype makes TNO believe that FAC simulation will be among the first applications of HMD 
based simulation systems that will come to an operational status. 

STINGER TEAM TRAINER 

Stinger is a man-portable, shoulder-fired, ground-to-air missile weapon system. It is mainly used to provide air 
defence against high-speed, low-level, ground-attack aircraft and helicopter. The system is operational with the Royal 
Netherlands armed forces (all three of Air Force, Army and Navy) and the armed forces of many other nations. 

Back in 1992, the TNO Physics and Electronics Laboratory developed a simple concept demonstrator that illustrates 
how HMD-based virtual environment technology can be applied to Stinger training (see Figure 4). It took until 1995 
for the Royal Netherlands Air Force to catch up with this concept and investigate whether a VE based simulator could 
benefit Stinger training. This resulted in a paperwork study by TNO that revealed the opportunities of a VE Stinger 
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trainer, but at the same time raised a number of unresolved questions with respect to the optimum design of such a VE 
Stinger trainer [4]. 

To answer the questions that have been raised by the paper work study in 1995, a new project has been planned to go 
on starting in 1998. The problem statement and planned approach of this project is described in the following sections. 



Figure 4. - The HMD based Stinger Trainer concept demonstrator as built by TNO in 1992 
(background image is image as seen in the HMD) 


Current Training Practice 

The Dutch armed forces currently have the following training devices available for Stinger training: 

• A dome-based trainer (see Figure 5) that provides a slide-projected terrain and up to two moving targets on a 20 
metres diameter dome; 

• A video-equiped training weapon system that is used in conjuction with live flying exercises; 

• The Stinger Troop Proficiency Trainer (STPT), a training weapon system with a small visor display that shows 
a virtual terrain with virtual targets. 

Novice Stinger personnel receive their initial training in the Stinger dome trainer. Combat readiness is then ensured in 
exercises with the video-based trainer and in live training at a shooting range. The STPT is supposed to be used for 
currency training (in practice, the STPT is not used at all). 

Training Bottlenecks 

The Royal Netherlands Air Force have indicated a requirement for better currency training tools. The new currency 
training tool shall: 

• Provide a broad range of training scenarios with complexity varying between novice level and extended air 
defence exercise levels; 

• Enable simulation of all Stinger part tasks; 

• Enable full team training; 

• Be mobile. 


None of the three training devices mentioned above fulfils these requirements. Subject of study is whether HMD- 
based virtual environment technology can do this job. An intrinsic issue here is that an HMD based Stinger trainer 
assures mobility while still providing a full 360 degrees field of view. 
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Problem Approach 

Based upon earlier work for Stinger training, the question whether a VE Stinger trainer is a feasible and valid solution 
to the Stinger training requirements has been reduced to first get answers on the following questions: 

a. To what specifications shall a mobile Stinger team training system adhere with respect to the following critical 
issues: 

- The type of display system; 

- The tracking system performance; 

- The end-to-end system latency. 

b. What is the expected learning transfer for a VE technology based mobile Stinger team training system? 



Figure 5. The Dutch armed forces currently use a dome-based training device for Stinger training 
Display System 

The most crucial and unclear aspect about the Stinger team training system is the way to setup the display system. We 
think of the following options to configure the display system: 

• Stereo immersive HMD; 

• Stereo see-through HMD; 

• Stereo see-through HMD combined with a visor display to obtain high precision target display; 

• Mono see-through HMD on the left eye combined with a visor display used by the right eye; 

• Projection screen; 

• Projection screen combined with a visor display to obtain high precision target display. 

The main issue to deal with here is the physical interaction between the head and the weapon system. In normal 
operation, the gunner will put his head against the visor. An HMD is a very elegant device to display a large 
surrounding environment with a yet small device, but it makes the physical contact between head and weapon system 
impossible. 

The latter three options listed above do enable the gunner to lay the head against the weapon. Most notably the 
configuration with a mono see-through HMD on the left eye and the visor display on the right eye is an interesting 
option to investigate. 
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Tracking System 

A tracking system is required to measure the position and orientation of both the HMD and the weapon. Given the 
small measures of the visor, this is a very accurate task. The tracking system shall therefore be: 

• Accurate; 

• Noise free; 

• Fast. 

It shall be determined what requirements are to be laid upon the performance of the tracking system with respect to 
these aspects. In addition to this, the characteristics of the tracking system can influence the operational restrictions of 
the training device. Some tracking systems put restrictions on the operating environment, or are very difficult to set up 
- this can conflict with the mobility requirement. 

System Latency 

Aiming a Stinger weapon system on a target is a perceptual motor task that can lead to unstable control in a simulator 
if end-to-end system latency is too large. The maximum latency that still provides acceptable simulation validity shall 
be determined. 

Learning Transfer 

The answers to the questions of how to setup the system design as posed in the previous sections are only of value if 
the VE simulator concept will be bring satisfactory learning transfer at all. The learning transfer with respect to the 
following Stinger tasks shall therefore be measured: 

• Target detection; 

• Target identification; 

• Aiming and firing; 

• Command and control processes; 

• Using weapon system peripherals; 

• Team operation. 

Evaluation Method 

The issues raised in the previous section will be dealt with by building an experimental Stinger team training system. 
The experimental system shall provide full functionality to simulate all relevant Stinger tasks and at the same time be 
flexible to enable the setup of different configurations with respect to the display system, the tracking performance and 
the end-to-end system latency. 

The experimental system will then be used for evaluations in three stages: 

• Component level evaluation; 

• Part task evaluation; 

• Integral evaluation. 

Component Level Evaluation 

The component level evaluation addresses the quality of the individual components used in the experimental system. 
The following aspects are investigated: 

• The quality of the display systems; 

• The quality of the tracking system; 

• The ability to position the head with respect to the weapon system; 

• The end-to-end system latency. 
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The results of this stage are required to interpret the results and identify involved causes in the remainder of the 
evaluations. 

Part Task Evaluation 

The part task evaluation aims at measuring human performance with respect to Stinger part tasks when using a specific 
configuration of the experimental Stinger trainer. Part task times and accuracies will be measured. 

Integral Evaluation 

The integral evaluation aims at measuring the learning transfer by determining human performance when performing a 
Stinger attack as a complete procedure. Hit-rates and procedure timing will be measured for a group of subjects on a 
typical series of Stinger training exercises. Part of the group will be subject to a number of different configurations of 
the experimental Stinger trainer. The subjects' exit level will be determined by going through a live training with video 
recording. 

CONCLUSION 

A HMD-based virtual environment system may provide a solution to the Stinger training problem, as the HMD 
provides a display system that provides a large field of regard, yet in a small, hence portable, device. The success of 
this system depends on a number of critical factors of which the display system and the corresponding physical 
interaction between the head and the weapon system is primary, then tracking performance and end-to-end system 
latency. The planned project will eliminate uncertainty on these factors. 

Conclusions 

As has been said in the introduction: The use of HMD technology in virtual environments for training simulation has 
always been recognised to be of great potential. So many have been proclaiming the infinite spectrum of applications 
that virtual environment technology will have in our daily lives. And so many are still waiting for real applications of 
this technology to appear. 

The Forward Air Controller training and Stinger team training applications in the military field have made us 
confident that real applications of HMD-based virtual environment systems do exist. The Forward Air Controller 
training study has proven that this type of system is accepted by the military users as a useful training tool. 
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